2024-03-25T16:11:34.444 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:11:34.444 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/orig.config.yaml 2024-03-25T16:11:34.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:11:34.566 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621255' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3t1f7dXKRia+bTNuo77Ot0iB2dDzXvpcv7bpKqPxqen4vit2GWi0a+v/UoPo9CIZ+qZGjp4U2gYFz1A6NfDAI= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAW6YOCv5MC8vYhC9glGx5t7q8Pdb+sqj/Fviye5coELMdd8qf1QHkf4B+qZAVgJRcyOxlZUIQBMF9DZM5dI6a4= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:11:34.567 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:11:34.567 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:11:34.568 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:11:34.569 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:11:34.598 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:11:34.598 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:11:34.598 INFO:teuthology.packaging:ref: None 2024-03-25T16:11:34.598 INFO:teuthology.packaging:tag: None 2024-03-25T16:11:34.598 INFO:teuthology.packaging:branch: main 2024-03-25T16:11:34.598 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:34.598 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:11:34.974 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:11:34.975 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:11:34.982 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:11:34.983 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:11:35.003 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:11:35.022 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:11:35.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:11:35.056 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.932853', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3t1f7dXKRia+bTNuo77Ot0iB2dDzXvpcv7bpKqPxqen4vit2GWi0a+v/UoPo9CIZ+qZGjp4U2gYFz1A6NfDAI='} 2024-03-25T16:11:35.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.934352', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAW6YOCv5MC8vYhC9glGx5t7q8Pdb+sqj/Fviye5coELMdd8qf1QHkf4B+qZAVgJRcyOxlZUIQBMF9DZM5dI6a4='} 2024-03-25T16:11:35.080 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:11:35.092 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:11:35.092 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:11:35.092 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:11:35.203 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc2accd70d0>, signals=[15]) 2024-03-25T16:11:35.203 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:11:35.211 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:11:35.211 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T16:11:35.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:35.305 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T16:11:35.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:35.396 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:11:35.404 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-03-25T16:11:35.422 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-03-25T16:11:35.422 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-03-25T16:11:35.478 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-03-25T16:11:35.478 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-03-25T16:11:35.478 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-03-25T16:11:35.478 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:11:35.478 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:11:35.479 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:11:35.480 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-03-25T16:11:35.507 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-03-25T16:11:35.524 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-03-25T16:11:35.524 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-03-25T16:11:35.579 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:11:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:11:35.580 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-03-25T16:11:35.599 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:11:35.608 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:11:35.626 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:11:35.627 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:11:35.629 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:11:35.646 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:11:35.676 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:11:35.676 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:11:35.685 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:11:35.991 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:11:36.046 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:11:36.060 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:11:36.060 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:11:36.061 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:11:36.061 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:11:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:11:36.078 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-25T16:11:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:11:36.079 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-25T16:11:36.135 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:11:36.135 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-361.el9.x86_64 2024-03-25T16:11:36.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-03-25T16:11:36.136 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:11:36.136 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-361.el9.x86_64 2024-03-25T16:11:36.136 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-03-25T16:11:38.311 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:11:38.437 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:11:39.820 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T16:11:39.936 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:11:41.011 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-03-25T16:11:41.470 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-25T16:11:44.194 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T16:11:49.069 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-25T16:11:50.669 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T16:11:52.640 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:11:57.570 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:11:59.677 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:11:59.677 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:11:59.780 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:11:59.792 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:11:59.793 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-03-25T16:11:59.793 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:11:59.793 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 76 M 2024-03-25T16:11:59.793 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 123 M 2024-03-25T16:11:59.793 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:12:00.346 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 157 kB/s | 44 kB 00:00 2024-03-25T16:12:01.822 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.5 MB/s | 15 MB 00:01 2024-03-25T16:12:02.971 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T16:12:03.496 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 8.7 MB/s | 27 MB 00:03 2024-03-25T16:12:03.500 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:03.500 INFO:teuthology.orchestra.run.smithi084.stdout:Total 20 MB/s | 76 MB 00:03 2024-03-25T16:12:04.165 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:12:04.521 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:12:04.521 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:12:05.519 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:12:05.519 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:12:07.975 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 744 kB/s | 19 MB 00:25 2024-03-25T16:12:07.975 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:12:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:11.411 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:11.541 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:14.384 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-25T16:12:15.826 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-25T16:12:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:17.738 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:12:20.982 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:22.685 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:12:24.825 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:12:24.825 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:12:24.920 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:12:24.926 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:12:24.927 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:12:24.927 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-03-25T16:12:24.927 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:12:24.927 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 76 M 2024-03-25T16:12:24.927 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-03-25T16:12:24.928 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:12:25.386 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 239 kB/s | 44 kB 00:00 2024-03-25T16:12:27.536 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.4 MB/s | 15 MB 00:02 2024-03-25T16:12:28.712 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 8.3 MB/s | 27 MB 00:03 2024-03-25T16:12:30.237 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.6 MB/s | 33 MB 00:05 2024-03-25T16:12:30.237 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:30.237 INFO:teuthology.orchestra.run.smithi157.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T16:12:30.907 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:12:31.318 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:12:31.318 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:12:32.348 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:12:32.349 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:12:34.846 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:12:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:38.347 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:38.491 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:43.228 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:43.424 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:48.211 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:50.609 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:50.609 INFO:teuthology.orchestra.run.smithi084.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:12:50.609 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:12:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:51.034 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:51.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:51.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:51.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:12:51.777 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:51.873 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.873 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:51.873 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:12:51.874 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:12:51.874 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:12:51.874 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:12:51.874 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:12:51.874 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:51.955 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:51.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:51.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-03-25T16:12:51.990 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-03-25T16:12:51.990 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-03-25T16:12:51.990 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-03-25T16:12:51.991 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 68.el9 2024-03-25T16:12:51.991 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-03-25T16:12:51.991 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:12:51.991 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-03-25T16:12:51.991 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8264254 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:12:51.992 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:12:51.993 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:12:51.994 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:12:51.995 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:12:51.995 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-03-25T16:12:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:12:51.995 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:12:51.997 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:12:51.997 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:12:51.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:52.636 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-03-25T16:12:54.009 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:54.051 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-03-25T16:12:54.054 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-25T16:12:54.076 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.Eb6uyHiQHV 2024-03-25T16:12:54.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Eb6uyHiQHV 2024-03-25T16:12:54.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.Eb6uyHiQHV 2024-03-25T16:12:54.308 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.Eb6uyHiQHV is 6KB 2024-03-25T16:12:54.337 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.Eb6uyHiQHV 2024-03-25T16:12:54.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:54.422 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:54.423 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:54.423 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:54.423 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:54.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:54.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-03-25T16:13:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:17.546 INFO:teuthology.orchestra.run.smithi157.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:13:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:13:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:17.922 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:17.923 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:13:17.923 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:13:17.923 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:13:18.290 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:13:18.291 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:13:18.601 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:18.735 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.735 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:18.735 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:13:18.735 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:13:18.735 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:13:18.736 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:13:18.736 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:13:18.736 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:18.815 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:18.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:18.816 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-03-25T16:13:18.851 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-03-25T16:13:18.851 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-03-25T16:13:18.851 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-03-25T16:13:18.851 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-03-25T16:13:18.852 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:13:18.853 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:13:18.854 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:13:18.854 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:13:18.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:13:19.471 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-03-25T16:13:20.745 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:13:20.777 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-03-25T16:13:20.780 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-25T16:13:20.795 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.16vRpmZAeT 2024-03-25T16:13:20.795 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.16vRpmZAeT 2024-03-25T16:13:20.864 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.16vRpmZAeT 2024-03-25T16:13:21.006 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.16vRpmZAeT is 6KB 2024-03-25T16:13:21.055 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.16vRpmZAeT 2024-03-25T16:13:21.069 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:13:21.135 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:13:21.135 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:13:21.135 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:13:21.135 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:13:21.136 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:13:21.289 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-03-25T16:13:24.582 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:13:24.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:13:24.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:43.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-25T16:13:51.298 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:13:51.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:13:51.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:52.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:13:52.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:55.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-25T16:14:07.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:14:07.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:09.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-25T16:14:10.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-25T16:14:18.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:14:18.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:21.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-25T16:14:25.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:14:25.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:28.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-25T16:14:33.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:14:33.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:36.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-25T16:14:46.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:14:46.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:46.649 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-25T16:14:47.027 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:14:47.027 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:47.028 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-25T16:14:47.041 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:47.042 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:47.042 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:47.042 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:48.043 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:14:48.043 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:48.116 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-03-25T16:14:48.150 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:51.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:14:51.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:15:10.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-25T16:15:28.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:15:28.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:15:28.664 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-25T16:15:29.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:15:29.040 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:15:29.040 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-25T16:15:29.054 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:15:29.054 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:15:29.054 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:15:29.054 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:15:30.055 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:15:30.056 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:15:30.134 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-03-25T16:15:30.177 DEBUG:teuthology.parallel:result is None 2024-03-25T16:15:30.178 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:15:30.186 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:15:30.186 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:15:30.189 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:15:30.205 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:15:30.212 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:15:30.222 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:15:30.222 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:15:30.247 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:15:30.339 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:15:30.347 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:15:30.347 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:15:30.350 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:15:30.392 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:30.396 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:30.406 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:30.409 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:30.411 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:15:30.418 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:15:30.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:15:30.421 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:15:30.449 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:15:30.459 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:15:30.459 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:15:30.491 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:15:30.506 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:30.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:30.643 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:30.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:30.770 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T16:15:30.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:15:30.837 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:30.879 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:30.933 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:31.014 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:31.058 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T16:15:31.058 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:15:31.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-03-25T16:15:31.128 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-03-25T16:15:31.168 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:15:31.213 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:15:31.714 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:15:31.722 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:15:31.722 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:15:31.802 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:15:31.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-03-25T16:15:31.853 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-03-25T16:15:31.855 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-03-25T16:15:31.909 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-03-25T16:15:31.911 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:15:31.911 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-03-25T16:15:31.929 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-03-25T16:15:31.929 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-03-25T16:15:31.953 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-03-25T16:15:31.953 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:15:31.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:15:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:31.996 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:15:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:32.025 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:15:32.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:15:32.072 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:15:32.105 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:15:32.114 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:15:32.133 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:15:32.134 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:19:50.373 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-03-25T16:19:50.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:19:50.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:50.466 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-25T16:19:50.548 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-25T16:19:50.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:19:50.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:50.639 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-25T16:19:50.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-25T16:19:50.714 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:19:50.725 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:19:50.725 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:50.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:50.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:50.728 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:50.757 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:50.773 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:50.784 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:50.801 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:50.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-03-25T16:19:50.816 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-03-25T16:19:50.830 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-03-25T16:19:50.833 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:50.844 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-03-25T16:19:50.849 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:50.860 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:50.877 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:50.903 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.907 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.932 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:50.935 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:50.935 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-25T16:19:50.935 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.935 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.936 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.936 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:50.936 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:19:50.944 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:19:50.944 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:19:50.944 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:19:50.945 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:19:50.950 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:19:50.950 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:19:50.951 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:50.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:50.953 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:50.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:51.179 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:51.179 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:51.183 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:51.184 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:51.507 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:51.507 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T16:19:51.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:51.536 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:51.536 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:51.572 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:51.572 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T16:19:51.572 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:51.601 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:51.602 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:51.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:51.671 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:51.683 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:51.750 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-03-25T16:19:51.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-03-25T16:19:51.752 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:51.821 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-03-25T16:19:51.822 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-03-25T16:19:52.008 INFO:teuthology.orchestra.run.smithi084.stdout:74 files removed 2024-03-25T16:19:52.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:52.077 INFO:teuthology.orchestra.run.smithi157.stdout:74 files removed 2024-03-25T16:19:52.105 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:53.254 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 89 kB/s | 82 kB 00:00 2024-03-25T16:19:53.506 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T16:19:54.096 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:19:54.297 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:19:54.963 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:19:55.097 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T16:19:55.557 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:19:55.558 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:19:56.982 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T16:19:57.442 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-03-25T16:19:58.424 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:20:06.096 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 936 kB/s | 8.0 MB 00:08 2024-03-25T16:20:10.414 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-25T16:20:14.723 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-03-25T16:20:16.965 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T16:20:18.741 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:20:20.923 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 899 kB/s | 19 MB 00:21 2024-03-25T16:20:23.607 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T16:20:24.998 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:20:26.035 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:20:26.039 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:20:26.040 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:20:26.041 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout:Install 30 Packages 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-03-25T16:20:26.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:20:26.043 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 58 M 2024-03-25T16:20:26.043 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:20:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-03-25T16:20:27.434 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:20:32.426 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:20:33.715 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 748 kB/s | 5.3 MB 00:07 2024-03-25T16:20:33.815 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:20:34.862 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:20:34.876 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:20:34.876 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:20:34.876 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:20:34.876 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:20:34.877 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:20:34.878 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-03-25T16:20:34.879 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:Install 30 Packages 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 58 M 2024-03-25T16:20:34.880 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:20:35.340 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 486 kB/s | 742 kB 00:01 2024-03-25T16:20:36.266 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 539 kB/s | 498 kB 00:00 2024-03-25T16:20:42.790 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 608 kB/s | 9.7 MB 00:16 2024-03-25T16:20:42.898 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 418 kB/s | 45 kB 00:00 2024-03-25T16:20:43.149 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-25T16:20:43.482 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 486 kB/s | 162 kB 00:00 2024-03-25T16:20:44.166 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 469 kB/s | 320 kB 00:00 2024-03-25T16:20:44.733 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 526 kB/s | 298 kB 00:00 2024-03-25T16:20:44.983 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-25T16:20:45.091 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T16:20:45.133 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:20:45.167 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:20:45.225 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:20:45.276 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-25T16:20:45.309 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:20:45.460 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-25T16:20:45.510 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-25T16:20:45.560 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 892 kB/s | 44 kB 00:00 2024-03-25T16:20:46.069 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 608 kB/s | 309 kB 00:00 2024-03-25T16:20:46.544 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 508 kB/s | 5.1 MB 00:10 2024-03-25T16:20:46.803 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-03-25T16:20:47.103 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-03-25T16:20:47.145 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-25T16:20:47.445 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-25T16:20:47.828 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-25T16:20:48.045 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 319 kB/s | 192 kB 00:00 2024-03-25T16:20:48.520 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-03-25T16:20:49.766 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 685 kB/s | 9.7 MB 00:14 2024-03-25T16:20:49.858 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:20:51.192 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 556 kB/s | 742 kB 00:01 2024-03-25T16:20:51.417 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 338 kB/s | 5.3 MB 00:16 2024-03-25T16:20:52.051 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 580 kB/s | 498 kB 00:00 2024-03-25T16:20:52.693 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 70 kB/s | 45 kB 00:00 2024-03-25T16:20:53.035 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 1.1 MB/s | 20 MB 00:17 2024-03-25T16:20:53.062 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 689 kB/s | 3.4 MB 00:05 2024-03-25T16:20:53.101 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 321 kB/s | 131 kB 00:00 2024-03-25T16:20:53.277 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 670 kB/s | 162 kB 00:00 2024-03-25T16:20:53.585 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 966 kB/s | 298 kB 00:00 2024-03-25T16:20:53.694 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 541 kB/s | 320 kB 00:00 2024-03-25T16:20:53.711 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 793 kB/s | 99 kB 00:00 2024-03-25T16:20:53.829 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 604 kB/s | 3.1 MB 00:05 2024-03-25T16:20:53.844 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-03-25T16:20:53.870 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T16:20:53.887 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:20:53.937 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:20:53.962 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T16:20:53.988 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-25T16:20:54.113 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 279 kB/s | 35 kB 00:00 2024-03-25T16:20:54.172 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:20:54.205 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 894 kB/s | 216 kB 00:00 2024-03-25T16:20:54.414 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T16:20:54.464 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-25T16:20:54.514 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 982 kB/s | 49 kB 00:00 2024-03-25T16:20:54.573 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-03-25T16:20:54.640 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-25T16:20:54.715 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-25T16:20:54.740 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T16:20:54.849 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-25T16:20:58.499 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 919 kB/s | 3.4 MB 00:03 2024-03-25T16:20:59.583 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 640 kB/s | 5.1 MB 00:08 2024-03-25T16:21:00.034 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 618 kB/s | 3.1 MB 00:05 2024-03-25T16:21:00.038 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:00.038 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.3 MB/s | 58 MB 00:25 2024-03-25T16:21:00.071 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:21:00.110 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:21:00.110 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:21:00.593 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:21:00.593 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:21:02.139 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:21:02.218 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:21:02.246 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:21:02.272 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:21:02.294 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:02.310 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:02.506 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:02.558 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:02.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:21:02.981 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:03.073 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:03.160 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:21:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:03.262 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:03.290 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:21:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:21:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:03.487 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:21:03.625 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:21:03.668 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:21:03.709 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:21:03.749 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:21:03.778 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:21:03.811 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:21:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:21:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:21:04.324 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:21:04.727 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 531 kB/s | 20 MB 00:38 2024-03-25T16:21:04.730 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:04.730 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.5 MB/s | 58 MB 00:38 2024-03-25T16:21:04.761 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:21:04.776 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:21:04.797 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:21:04.797 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:21:04.802 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:04.873 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:04.950 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:21:04.979 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:21:05.264 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:21:05.265 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:21:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:21:06.829 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:21:06.865 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:21:06.910 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:21:06.938 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:06.954 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:07.134 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:07.176 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:07.530 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:21:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:07.716 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:07.754 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:21:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:07.852 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:07.852 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:21:07.852 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:21:07.853 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:07.876 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:21:07.897 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:07.911 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:21:08.131 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:08.170 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:08.233 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:08.260 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:21:08.368 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:21:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:21:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:21:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:21:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:21:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:21:08.845 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:21:08.876 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:08.950 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:08.981 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:21:09.024 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:21:09.483 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:21:09.512 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:09.566 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:09.625 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:21:09.656 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:21:11.770 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:11.802 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:12.348 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:12.388 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:12.481 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:12.481 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:21:12.481 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:21:12.482 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:12.515 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-03-25T16:21:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:17.987 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:18.011 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:18.013 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:18.082 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:18.082 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:19.171 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:19.171 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:21:19.171 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:19.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:21:19.173 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:21:19.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:21:19.175 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:21:19.176 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:21:19.176 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:21:19.176 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:21:19.177 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:19.177 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:21:19.177 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:21:20.189 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:20.189 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:21:20.190 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:21:20.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:20.192 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:21:20.517 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-03-25T16:21:21.082 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:49 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:21:21.235 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:21:21.236 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:21:21.236 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:21:21.236 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:21:21.236 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-25T16:21:21.237 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:21:21.238 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 48 M 2024-03-25T16:21:21.238 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 204 M 2024-03-25T16:21:21.238 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:21:21.464 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-25T16:21:21.573 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.7 MB/s | 300 kB 00:00 2024-03-25T16:21:21.606 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-03-25T16:21:21.723 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-03-25T16:21:21.724 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:21.818 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 902 kB/s | 187 kB 00:00 2024-03-25T16:21:22.265 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 178 kB/s | 163 kB 00:00 2024-03-25T16:21:22.635 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:22.659 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:22.659 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:22.660 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:21:22.660 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:22.660 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:22.660 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:22.661 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:22.755 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:22.755 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:21:23.819 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:21:23.820 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:21:23.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:21:23.822 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:21:23.822 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:23.822 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:21:23.822 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:21:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:24.876 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:21:25.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-03-25T16:21:25.787 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:02 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:21:25.934 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:21:25.934 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:21:25.934 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:21:25.934 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-25T16:21:25.935 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:21:25.936 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 48 M 2024-03-25T16:21:25.936 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 204 M 2024-03-25T16:21:25.936 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:21:26.220 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 902 kB/s | 217 kB 00:00 2024-03-25T16:21:26.312 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-25T16:21:26.354 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:21:26.479 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:21:26.704 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 225 kB/s | 163 kB 00:00 2024-03-25T16:22:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 832 kB/s | 47 MB 00:58 2024-03-25T16:22:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:Total 847 kB/s | 48 MB 00:58 2024-03-25T16:22:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:22:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:22:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:22:24.595 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:22:24.595 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:22:25.261 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:22:25.311 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:22:25.351 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:22:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:22:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:22:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:22:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:22:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:22:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:22:32.700 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:22:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:22:32.701 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:22:32.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-03-25T16:22:33.563 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:10 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:22:33.721 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:22:33.723 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-25T16:22:33.723 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:22:33.723 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-25T16:22:33.723 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:22:33.724 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:22:33.725 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:22:33.726 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout:Install 36 Packages 2024-03-25T16:22:33.727 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:22:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2024-03-25T16:22:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 107 M 2024-03-25T16:22:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:22:34.370 INFO:teuthology.orchestra.run.smithi084.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:22:34.614 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 661 kB/s | 47 MB 01:13 2024-03-25T16:22:34.616 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:34.616 INFO:teuthology.orchestra.run.smithi157.stdout:Total 672 kB/s | 48 MB 01:13 2024-03-25T16:22:34.628 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:22:34.646 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:22:34.647 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:22:35.029 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:22:35.029 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:22:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:22:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:22:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:22:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:22:35.875 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:36.061 INFO:teuthology.orchestra.run.smithi084.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 718 kB/s | 1.5 MB 00:02 2024-03-25T16:22:36.369 INFO:teuthology.orchestra.run.smithi084.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 894 kB/s | 2.1 MB 00:02 2024-03-25T16:22:36.595 INFO:teuthology.orchestra.run.smithi084.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.1 MB/s | 247 kB 00:00 2024-03-25T16:22:36.762 INFO:teuthology.orchestra.run.smithi084.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-25T16:22:36.803 INFO:teuthology.orchestra.run.smithi084.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:22:36.829 INFO:teuthology.orchestra.run.smithi084.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:22:36.854 INFO:teuthology.orchestra.run.smithi084.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:22:36.904 INFO:teuthology.orchestra.run.smithi084.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-25T16:22:36.946 INFO:teuthology.orchestra.run.smithi084.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:22:37.013 INFO:teuthology.orchestra.run.smithi084.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi084.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 103 kB 00:00 2024-03-25T16:22:37.097 INFO:teuthology.orchestra.run.smithi084.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:22:37.164 INFO:teuthology.orchestra.run.smithi084.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.8 MB/s | 187 kB 00:00 2024-03-25T16:22:37.373 INFO:teuthology.orchestra.run.smithi084.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 599 kB/s | 125 kB 00:00 2024-03-25T16:22:37.540 INFO:teuthology.orchestra.run.smithi084.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-25T16:22:37.882 INFO:teuthology.orchestra.run.smithi084.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 442 kB/s | 151 kB 00:00 2024-03-25T16:22:37.966 INFO:teuthology.orchestra.run.smithi084.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 522 kB/s | 43 kB 00:00 2024-03-25T16:22:38.074 INFO:teuthology.orchestra.run.smithi084.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-25T16:22:38.216 INFO:teuthology.orchestra.run.smithi084.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-25T16:22:38.283 INFO:teuthology.orchestra.run.smithi084.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-25T16:22:38.350 INFO:teuthology.orchestra.run.smithi084.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T16:22:38.417 INFO:teuthology.orchestra.run.smithi084.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T16:22:38.509 INFO:teuthology.orchestra.run.smithi084.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T16:22:38.576 INFO:teuthology.orchestra.run.smithi084.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 296 kB/s | 20 kB 00:00 2024-03-25T16:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-25T16:22:38.718 INFO:teuthology.orchestra.run.smithi084.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T16:22:38.810 INFO:teuthology.orchestra.run.smithi084.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-25T16:22:38.893 INFO:teuthology.orchestra.run.smithi084.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-03-25T16:22:38.969 INFO:teuthology.orchestra.run.smithi084.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:22:39.035 INFO:teuthology.orchestra.run.smithi084.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 538 kB/s | 36 kB 00:00 2024-03-25T16:22:39.136 INFO:teuthology.orchestra.run.smithi084.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T16:22:39.244 INFO:teuthology.orchestra.run.smithi084.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-25T16:22:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T16:22:40.373 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:40.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:42.719 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:42.720 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:22:42.720 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:22:42.720 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:42.720 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:22:42.720 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:22:42.736 INFO:teuthology.orchestra.run.smithi084.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 573 kB/s | 4.7 MB 00:08 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:22:43.093 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:22:43.423 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-03-25T16:22:43.989 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:22:44.143 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:22:44.145 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:22:44.146 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:22:44.147 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:22:44.149 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:22:44.150 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout:Install 36 Packages 2024-03-25T16:22:44.151 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:22:44.152 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 30 M 2024-03-25T16:22:44.152 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 107 M 2024-03-25T16:22:44.152 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:22:44.649 INFO:teuthology.orchestra.run.smithi157.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:22:46.149 INFO:teuthology.orchestra.run.smithi157.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 812 kB/s | 1.5 MB 00:01 2024-03-25T16:22:47.166 INFO:teuthology.orchestra.run.smithi157.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 751 kB/s | 2.1 MB 00:02 2024-03-25T16:22:47.500 INFO:teuthology.orchestra.run.smithi157.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 743 kB/s | 247 kB 00:00 2024-03-25T16:22:47.816 INFO:teuthology.orchestra.run.smithi157.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 590 kB/s | 187 kB 00:00 2024-03-25T16:22:47.892 INFO:teuthology.orchestra.run.smithi157.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-25T16:22:47.967 INFO:teuthology.orchestra.run.smithi157.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 421 kB/s | 32 kB 00:00 2024-03-25T16:22:48.009 INFO:teuthology.orchestra.run.smithi157.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:22:48.042 INFO:teuthology.orchestra.run.smithi157.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:22:48.093 INFO:teuthology.orchestra.run.smithi157.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:22:48.193 INFO:teuthology.orchestra.run.smithi157.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:22:48.227 INFO:teuthology.orchestra.run.smithi157.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:22:48.260 INFO:teuthology.orchestra.run.smithi157.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:22:48.297 INFO:teuthology.orchestra.run.smithi157.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:22:48.446 INFO:teuthology.orchestra.run.smithi157.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 833 kB/s | 125 kB 00:00 2024-03-25T16:22:48.486 INFO:teuthology.orchestra.run.smithi157.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 538 kB/s | 22 kB 00:00 2024-03-25T16:22:48.628 INFO:teuthology.orchestra.run.smithi157.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-25T16:22:48.670 INFO:teuthology.orchestra.run.smithi157.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-25T16:22:48.728 INFO:teuthology.orchestra.run.smithi157.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-25T16:22:48.812 INFO:teuthology.orchestra.run.smithi157.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-03-25T16:22:48.846 INFO:teuthology.orchestra.run.smithi157.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-03-25T16:22:48.879 INFO:teuthology.orchestra.run.smithi157.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2024-03-25T16:22:48.913 INFO:teuthology.orchestra.run.smithi157.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 697 kB/s | 23 kB 00:00 2024-03-25T16:22:48.947 INFO:teuthology.orchestra.run.smithi157.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 585 kB/s | 19 kB 00:00 2024-03-25T16:22:48.980 INFO:teuthology.orchestra.run.smithi157.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 593 kB/s | 20 kB 00:00 2024-03-25T16:22:49.014 INFO:teuthology.orchestra.run.smithi157.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-03-25T16:22:49.056 INFO:teuthology.orchestra.run.smithi157.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-25T16:22:49.106 INFO:teuthology.orchestra.run.smithi157.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:22:49.140 INFO:teuthology.orchestra.run.smithi157.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 495 kB/s | 16 kB 00:00 2024-03-25T16:22:49.182 INFO:teuthology.orchestra.run.smithi157.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-25T16:22:49.224 INFO:teuthology.orchestra.run.smithi157.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-03-25T16:22:49.275 INFO:teuthology.orchestra.run.smithi157.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T16:22:49.333 INFO:teuthology.orchestra.run.smithi157.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-25T16:22:49.366 INFO:teuthology.orchestra.run.smithi157.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 603 kB/s | 20 kB 00:00 2024-03-25T16:22:52.199 INFO:teuthology.orchestra.run.smithi157.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 635 kB/s | 4.7 MB 00:07 2024-03-25T16:22:58.117 INFO:teuthology.orchestra.run.smithi084.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 779 kB/s | 17 MB 00:22 2024-03-25T16:22:58.123 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:58.123 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.2 MB/s | 30 MB 00:24 2024-03-25T16:22:58.147 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:22:58.188 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:22:58.189 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:22:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:22:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:22:59.922 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:22:59.985 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:23:00.023 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:23:00.054 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:23:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:23:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:23:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:23:00.295 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:23:00.374 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:23:00.422 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:23:00.494 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:23:00.547 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:23:00.586 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:23:00.635 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:23:00.689 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:23:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:23:00.923 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:23:00.962 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:23:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:23:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:23:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:23:01.128 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:01.161 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:23:01.201 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:23:01.236 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:23:01.271 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:23:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:23:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:23:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:23:01.612 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:23:01.743 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:01.901 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:23:01.931 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:01.956 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:01.956 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:01.956 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:23:01.957 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:01.957 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:01.957 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:02.773 INFO:teuthology.orchestra.run.smithi157.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-25T16:23:02.777 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:02.777 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.6 MB/s | 30 MB 00:18 2024-03-25T16:23:02.802 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:02.844 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:02.844 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:03.243 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:03.244 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:03.390 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:03.416 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:03.417 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:03.417 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:23:03.417 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:03.417 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:03.417 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:03.858 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:03.881 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:03.881 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:03.881 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:23:03.881 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:03.881 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:03.882 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:04.092 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:04.114 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:04.115 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:04.115 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:23:04.115 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:04.115 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:04.115 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:04.180 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:04.588 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:04.659 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:23:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:23:04.719 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:23:04.799 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:23:04.843 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:23:04.876 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:23:04.968 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:23:05.055 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:23:05.095 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:23:05.167 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:23:05.221 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:23:05.259 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:23:05.308 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:23:05.363 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:23:05.549 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:23:05.597 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:23:05.635 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:23:05.678 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:23:05.708 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:23:05.754 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:05.754 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:23:05.754 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:23:05.754 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:23:05.755 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:23:05.756 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:23:05.757 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:23:05.757 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:23:05.757 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:23:05.785 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:23:05.818 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:05.860 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:23:05.901 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:23:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:23:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:23:06.068 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:23:06.109 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:23:06.182 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:23:06.319 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:23:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:06.602 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:23:06.631 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:06.655 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:06.656 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:06.656 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:23:06.656 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:06.656 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:06.656 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:06.848 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:23:06.849 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:23:06.850 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:06.851 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:07.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-03-25T16:23:07.778 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:07.874 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:07.926 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:07.926 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:23:07.927 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:07.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-03-25T16:23:08.082 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:08.108 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:08.544 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:08.549 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:08.572 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:08.691 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:23:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 747 k 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 754 k 2024-03-25T16:23:08.693 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:08.791 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:08.814 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:08.815 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:08.815 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:23:08.815 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:08.815 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:08.815 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:09.871 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 635 kB/s | 747 kB 00:01 2024-03-25T16:23:09.871 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:09.872 INFO:teuthology.orchestra.run.smithi084.stdout:Total 634 kB/s | 747 kB 00:01 2024-03-25T16:23:09.872 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:09.878 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:09.878 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:09.884 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:09.885 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:09.964 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:10.397 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:10.492 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:10.617 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:10.617 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:23:10.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:23:10.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:23:10.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:23:10.871 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:11.177 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:11.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:23:11.525 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:23:11.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:23:11.526 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:23:11.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:11.528 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:11.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-03-25T16:23:11.946 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:12.100 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 145 k 2024-03-25T16:23:12.101 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 436 k 2024-03-25T16:23:12.102 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:12.385 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:40 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:12.488 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:12.543 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:12.544 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:23:12.544 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:12.621 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-03-25T16:23:12.850 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 194 kB/s | 145 kB 00:00 2024-03-25T16:23:12.851 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:12.851 INFO:teuthology.orchestra.run.smithi084.stdout:Total 194 kB/s | 145 kB 00:00 2024-03-25T16:23:12.852 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:12.859 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:12.860 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:12.912 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:12.913 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:13.050 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:13.101 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:13.125 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:13.126 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:13.126 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:23:13.126 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:13.193 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:13.342 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:13.343 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:13.343 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:13.343 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:13.343 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 747 k 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 754 k 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:14.073 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:14.073 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:14.074 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:14.074 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:14.074 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:14.074 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:14.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-03-25T16:23:14.782 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 520 kB/s | 747 kB 00:01 2024-03-25T16:23:14.782 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:14.783 INFO:teuthology.orchestra.run.smithi157.stdout:Total 519 kB/s | 747 kB 00:01 2024-03-25T16:23:14.783 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:14.789 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:14.789 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:14.795 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:14.796 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:14.871 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:14.905 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:51 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:15.004 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:15.058 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:15.059 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:23:15.059 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:15.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:23:15.327 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:15.411 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:15.676 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:15.793 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:15.824 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:15.825 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-25T16:23:15.825 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:23:15.826 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:15.827 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.9 M 2024-03-25T16:23:15.828 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-25T16:23:15.828 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:16.077 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:16.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:23:16.478 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T16:23:16.720 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-03-25T16:23:16.753 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 43 kB/s | 31 kB 00:00 2024-03-25T16:23:16.779 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.0 MB/s | 238 kB 00:00 2024-03-25T16:23:16.866 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:17.020 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:17.021 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:23:17.021 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:17.021 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 145 k 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 436 k 2024-03-25T16:23:17.022 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:17.037 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T16:23:17.204 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:23:17.271 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-25T16:23:17.438 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.7 MB/s | 2.0 MB 00:00 2024-03-25T16:23:17.463 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 752 kB/s | 144 kB 00:00 2024-03-25T16:23:17.505 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 459 kB/s | 31 kB 00:00 2024-03-25T16:23:17.589 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-25T16:23:17.838 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 178 kB/s | 145 kB 00:00 2024-03-25T16:23:17.838 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:17.838 INFO:teuthology.orchestra.run.smithi157.stdout:Total 178 kB/s | 145 kB 00:00 2024-03-25T16:23:17.838 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:17.847 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:17.847 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:17.901 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:17.901 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:18.043 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:18.105 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:18.130 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:18.130 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:18.130 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:23:18.130 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:18.780 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 660 kB/s | 1.8 MB 00:02 2024-03-25T16:23:18.786 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:18.786 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.3 MB/s | 6.9 MB 00:02 2024-03-25T16:23:18.806 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:18.825 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:18.825 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:18.990 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:18.991 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:19.358 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-03-25T16:23:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:19.609 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:23:19.649 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:23:19.695 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:23:19.735 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:23:19.840 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:23:19.938 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:47 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:19.977 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:23:20.035 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:23:20.087 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:20.088 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:23:20.088 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:23:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:20.168 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:23:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:23:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:23:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:20.733 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:20.882 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:23:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-25T16:23:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:20.885 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.9 M 2024-03-25T16:23:20.885 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-25T16:23:20.885 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:23:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:23:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:23:21.600 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 69 kB/s | 31 kB 00:00 2024-03-25T16:23:21.625 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:23:21.638 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:21.639 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:21.640 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:21.784 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-25T16:23:21.817 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:23:21.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:23:22.109 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-25T16:23:22.451 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 57 kB/s | 19 kB 00:00 2024-03-25T16:23:22.474 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:22.493 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 812 kB/s | 548 kB 00:00 2024-03-25T16:23:22.618 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:23:22.627 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:22.628 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-25T16:23:22.628 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:23:22.628 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-25T16:23:22.628 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:23:22.629 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 209 M 2024-03-25T16:23:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:22.727 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T16:23:22.844 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:23:22.861 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 272 kB/s | 32 kB 00:00 2024-03-25T16:23:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 663 kB/s | 17 kB 00:00 2024-03-25T16:23:22.945 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:23:23.077 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-25T16:23:23.104 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:23:23.229 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 39 MB/s | 4.8 MB 00:00 2024-03-25T16:23:23.270 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T16:23:23.438 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-03-25T16:23:23.646 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T16:23:23.871 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-25T16:23:23.896 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-03-25T16:23:24.047 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-25T16:23:24.986 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 475 kB/s | 1.8 MB 00:03 2024-03-25T16:23:24.992 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:24.992 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.7 MB/s | 6.9 MB 00:04 2024-03-25T16:23:25.013 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:25.032 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:25.032 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:25.220 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:25.220 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:25.828 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:23:25.871 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:23:25.918 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:23:25.949 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:23:26.045 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:23:26.173 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:23:26.267 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:23:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:23:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:23:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:23:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:27.411 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:27.411 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:23:27.411 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:23:27.411 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:23:27.411 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:23:27.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:23:27.759 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:23:27.759 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:27.759 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:27.759 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:23:27.759 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:27.760 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:28.032 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:23:28.598 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:56 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:28.748 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:23:28.749 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-25T16:23:28.750 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:28.751 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 37 M 2024-03-25T16:23:28.751 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 209 M 2024-03-25T16:23:28.751 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:29.209 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 286 kB/s | 32 kB 00:00 2024-03-25T16:23:29.258 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 655 kB/s | 17 kB 00:00 2024-03-25T16:23:29.284 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:23:29.451 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T16:23:29.585 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:23:29.634 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T16:23:29.763 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2024-03-25T16:23:30.019 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T16:23:30.094 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-25T16:23:30.153 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-25T16:23:30.969 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 280 kB/s | 245 kB 00:00 2024-03-25T16:23:39.693 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 714 kB/s | 7.4 MB 00:10 2024-03-25T16:23:39.696 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:39.696 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.4 MB/s | 37 MB 00:10 2024-03-25T16:23:39.741 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:39.767 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:39.767 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:40.130 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:40.130 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:23:40.955 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:23:41.053 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:41.109 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:23:41.287 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:41.316 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:23:41.485 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 404 kB/s | 7.4 MB 00:18 2024-03-25T16:23:41.487 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:41.488 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.0 MB/s | 37 MB 00:18 2024-03-25T16:23:41.530 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:41.556 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:41.556 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:41.930 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:41.930 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:42.050 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:23:42.179 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:23:42.643 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:42.716 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:23:42.747 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:23:42.848 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:42.884 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:23:43.063 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:23:43.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:23:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:23:43.987 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:23:44.452 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:23:44.490 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:45.210 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:23:46.223 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:23:46.256 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:47.092 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:47.092 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:23:47.092 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:23:47.092 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:47.092 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:23:47.093 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:23:47.094 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:23:47.491 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:23:47.491 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:23:47.492 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:47.493 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:47.876 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:23:48.449 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:48.601 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:23:48.603 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:23:48.604 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:23:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:48.606 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:48.606 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:48.606 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-03-25T16:23:48.606 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:48.607 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2024-03-25T16:23:48.607 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-25T16:23:48.608 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:48.730 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:48.730 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:23:48.730 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:23:48.730 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:23:48.731 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:23:48.858 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 238 kB/s | 24 kB 00:00 2024-03-25T16:23:48.908 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-25T16:23:48.958 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:23:48.983 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:23:49.008 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:23:49.034 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 869 kB/s | 43 kB 00:00 2024-03-25T16:23:49.113 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:23:49.113 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:49.113 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:23:49.114 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:23:49.115 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:49.115 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:49.115 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:23:49.115 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:49.115 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:49.301 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-25T16:23:49.326 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 102 kB/s | 32 kB 00:00 2024-03-25T16:23:49.384 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 83 kB/s | 52 kB 00:00 2024-03-25T16:23:49.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:23:49.726 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-25T16:23:49.752 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 450 kB/s | 203 kB 00:00 2024-03-25T16:23:50.002 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-25T16:23:50.054 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:50.069 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 262 kB/s | 90 kB 00:00 2024-03-25T16:23:50.071 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:50.072 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-25T16:23:50.072 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:50.087 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:50.087 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:50.181 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:50.183 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:50.202 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:23:50.204 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-03-25T16:23:50.205 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:50.206 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.0 M 2024-03-25T16:23:50.206 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-25T16:23:50.206 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:50.639 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 262 kB/s | 24 kB 00:00 2024-03-25T16:23:50.690 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-25T16:23:50.701 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:50.739 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:23:50.765 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:23:50.790 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:23:50.815 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:23:50.868 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:23:50.899 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 180 kB/s | 15 kB 00:00 2024-03-25T16:23:50.924 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2024-03-25T16:23:50.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:23:50.992 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:23:51.016 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.7 MB/s | 203 kB 00:00 2024-03-25T16:23:51.029 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:23:51.066 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T16:23:51.070 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:23:51.108 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-03-25T16:23:51.133 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T16:23:51.164 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:23:51.167 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 84 kB/s | 52 kB 00:00 2024-03-25T16:23:51.168 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:51.169 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T16:23:51.169 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:51.182 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:51.182 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:51.272 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:51.273 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:51.282 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:23:51.330 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:23:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:51.948 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:23:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:23:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:23:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:23:52.111 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:52.127 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:23:52.144 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:52.193 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:52.220 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:23:52.321 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:23:52.379 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:23:53.020 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:23:53.022 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:23:53.022 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:23:53.022 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:23:53.022 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:23:53.022 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:23:53.023 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:23:53.106 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:53.158 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:53.192 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:53.240 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:53.262 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:53.380 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:23:53.381 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:23:53.382 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:23:53.382 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:53.382 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:53.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:23:54.111 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:23:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:23:54.207 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:22 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:54.358 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:23:54.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:54.360 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.8 M 2024-03-25T16:23:54.361 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-03-25T16:23:54.361 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:54.485 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:23:54.486 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:23:54.487 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:23:54.487 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:23:54.487 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:54.487 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:54.679 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-25T16:23:54.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:23:54.763 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:23:54.896 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 434 kB/s | 58 kB 00:00 2024-03-25T16:23:54.955 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 872 kB/s | 51 kB 00:00 2024-03-25T16:23:54.988 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:23:55.188 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 193 kB/s | 138 kB 00:00 2024-03-25T16:23:55.190 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:55.190 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.2 MB/s | 6.8 MB 00:00 2024-03-25T16:23:55.203 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:23:55.214 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:23:55.214 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:23:55.302 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:23:55.304 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:23:55.312 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:23:55.462 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:23:55.463 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:23:55.463 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:55.463 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:23:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-25T16:23:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:55.465 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.8 M 2024-03-25T16:23:55.465 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 33 M 2024-03-25T16:23:55.465 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:23:55.604 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:23:55.789 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:55.840 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:56.234 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:23:56.293 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 650 kB/s | 248 kB 00:00 2024-03-25T16:23:56.363 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:56.385 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-03-25T16:23:56.410 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 494 kB/s | 58 kB 00:00 2024-03-25T16:23:56.435 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T16:23:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:56.513 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:56.544 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:56.644 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 188 kB/s | 138 kB 00:00 2024-03-25T16:23:56.645 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:56.645 INFO:teuthology.orchestra.run.smithi084.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-25T16:23:56.658 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:23:56.668 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:23:56.668 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:23:56.757 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:23:56.758 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:23:57.053 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:23:57.227 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:57.281 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:57.355 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:57.355 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:57.356 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:57.356 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:57.356 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:57.356 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:57.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:57.646 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:57.646 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:57.647 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:23:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:57.883 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:57.896 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-03-25T16:23:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:58.476 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:23:58.628 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:23:58.628 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:58.628 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:23:58.628 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-25T16:23:58.629 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:23:58.630 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 964 k 2024-03-25T16:23:58.630 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.8 M 2024-03-25T16:23:58.630 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:23:58.746 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:58.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:58.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:58.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:58.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:58.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:58.789 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 493 kB/s | 79 kB 00:00 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:23:59.041 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:23:59.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-03-25T16:23:59.856 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:36 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:00.005 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 964 k 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.8 M 2024-03-25T16:24:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:00.154 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 536 kB/s | 79 kB 00:00 2024-03-25T16:24:00.615 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 446 kB/s | 885 kB 00:01 2024-03-25T16:24:00.616 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:00.616 INFO:teuthology.orchestra.run.smithi157.stdout:Total 485 kB/s | 964 kB 00:01 2024-03-25T16:24:00.617 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:00.640 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:00.640 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:00.718 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:00.719 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:00.918 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:01.065 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:24:01.139 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:01.721 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 516 kB/s | 885 kB 00:01 2024-03-25T16:24:01.722 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:01.723 INFO:teuthology.orchestra.run.smithi084.stdout:Total 561 kB/s | 964 kB 00:01 2024-03-25T16:24:01.723 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:01.747 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:01.747 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:01.805 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:01.806 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:24:01.824 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:01.824 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:02.010 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:02.112 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:24:02.112 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:02.113 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:02.113 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:24:02.113 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:02.113 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:02.155 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:24:02.231 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:02.379 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-03-25T16:24:02.916 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:02.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:24:02.962 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:03.113 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-25T16:24:03.114 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:03.115 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 332 k 2024-03-25T16:24:03.115 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.5 M 2024-03-25T16:24:03.115 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:24:03.203 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:03.203 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:03.271 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 442 kB/s | 69 kB 00:00 2024-03-25T16:24:03.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-volume 2024-03-25T16:24:04.056 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:41 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:04.211 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:04.212 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:04.213 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:04.213 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-25T16:24:04.213 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:04.214 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 332 k 2024-03-25T16:24:04.214 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.5 M 2024-03-25T16:24:04.214 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:04.254 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 232 kB/s | 264 kB 00:01 2024-03-25T16:24:04.256 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:04.256 INFO:teuthology.orchestra.run.smithi157.stdout:Total 291 kB/s | 332 kB 00:01 2024-03-25T16:24:04.256 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:04.266 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:04.266 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:04.308 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:04.308 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:04.366 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 450 kB/s | 69 kB 00:00 2024-03-25T16:24:04.458 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:04.646 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:24:04.690 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:04.706 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:04.706 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:04.706 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:24:04.706 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:05.091 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 300 kB/s | 264 kB 00:00 2024-03-25T16:24:05.092 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:05.093 INFO:teuthology.orchestra.run.smithi084.stdout:Total 378 kB/s | 332 kB 00:00 2024-03-25T16:24:05.093 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:05.101 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:05.102 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:05.139 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:05.139 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:05.283 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:05.464 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:24:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:24:05.510 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:05.526 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:05.526 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:05.526 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:24:05.526 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:05.803 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:05.804 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:06.144 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-03-25T16:24:06.255 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:24:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:06.628 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:06.724 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:34 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:06.874 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 127 k 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 456 k 2024-03-25T16:24:06.876 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:06.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-03-25T16:24:07.572 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:07.720 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:24:07.721 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:07.722 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 127 k 2024-03-25T16:24:07.722 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2024-03-25T16:24:07.722 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:07.742 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 146 kB/s | 127 kB 00:00 2024-03-25T16:24:07.743 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:07.743 INFO:teuthology.orchestra.run.smithi157.stdout:Total 146 kB/s | 127 kB 00:00 2024-03-25T16:24:07.744 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:07.750 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:07.750 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:07.812 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:07.812 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:07.960 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:08.075 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:08.523 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:08.571 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 149 kB/s | 127 kB 00:00 2024-03-25T16:24:08.572 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:08.572 INFO:teuthology.orchestra.run.smithi084.stdout:Total 149 kB/s | 127 kB 00:00 2024-03-25T16:24:08.572 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:08.578 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:08.578 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:08.633 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:08.634 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:08.792 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:08.792 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:08.793 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:08.793 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:08.793 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:08.793 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:08.878 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:09.012 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-03-25T16:24:09.303 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:09.597 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:09.600 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:09.696 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:09.750 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:09.751 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:09.751 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:09.834 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-03-25T16:24:09.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-03-25T16:24:10.403 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:10.428 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:10.528 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:10.552 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:24:10.553 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 31 k 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 140 k 2024-03-25T16:24:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:10.582 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:10.583 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:10.583 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:10.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-03-25T16:24:11.087 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T16:24:11.088 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:11.088 INFO:teuthology.orchestra.run.smithi157.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T16:24:11.088 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:11.092 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:11.092 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:11.120 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:11.121 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:11.217 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:48 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:11.264 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:11.366 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 k 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 140 k 2024-03-25T16:24:11.367 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:11.377 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:11.766 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:11.882 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 61 kB/s | 31 kB 00:00 2024-03-25T16:24:11.883 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:11.883 INFO:teuthology.orchestra.run.smithi084.stdout:Total 60 kB/s | 31 kB 00:00 2024-03-25T16:24:11.883 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:11.886 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:11.886 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:11.913 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:11.914 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:12.027 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:12.028 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:12.028 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:12.028 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:12.028 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:12.028 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:12.046 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:12.138 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:12.168 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-03-25T16:24:12.532 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:12.740 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:40 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:12.784 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:12.784 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:12.784 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:12.784 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:12.784 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:12.785 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:12.838 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:12.890 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:12.891 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:12.891 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:12.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-03-25T16:24:12.949 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-03-25T16:24:13.510 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:13.518 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:41 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:13.608 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:13.615 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:13.660 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:13.661 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:13.661 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:13.666 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:13.667 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:13.668 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:13.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-03-25T16:24:13.723 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-03-25T16:24:14.277 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:51 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:14.298 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:42 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:14.373 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:14.400 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:14.425 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:14.426 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:14.426 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:14.456 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:14.457 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:14.457 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:14.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-03-25T16:24:14.511 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-03-25T16:24:15.033 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:15.107 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:15.130 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:15.182 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:15.183 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:15.183 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:15.206 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:15.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-03-25T16:24:15.258 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:15.259 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:15.259 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:15.304 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-03-25T16:24:15.795 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:15.889 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:15.890 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:15.942 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:15.943 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:15.943 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:15.985 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:15.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-03-25T16:24:16.036 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:16.037 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:16.037 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:16.091 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-03-25T16:24:16.564 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:53 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:16.660 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:16.661 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:16.712 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:16.713 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:16.714 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:16.758 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:16.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-03-25T16:24:16.809 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:16.810 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T16:24:16.810 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:16.866 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-03-25T16:24:17.365 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:17.436 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:45 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:17.466 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:17.520 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:17.521 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T16:24:17.521 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:17.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-03-25T16:24:17.586 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:17.588 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 85 k 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 231 k 2024-03-25T16:24:17.589 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:18.164 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:18.182 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 144 kB/s | 85 kB 00:00 2024-03-25T16:24:18.183 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:18.183 INFO:teuthology.orchestra.run.smithi157.stdout:Total 143 kB/s | 85 kB 00:00 2024-03-25T16:24:18.183 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:18.191 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:18.191 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:18.249 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:18.249 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:18.314 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 85 k 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 231 k 2024-03-25T16:24:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:18.489 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:18.900 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.026 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 85 kB 00:00 2024-03-25T16:24:19.027 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:19.027 INFO:teuthology.orchestra.run.smithi084.stdout:Total 119 kB/s | 85 kB 00:00 2024-03-25T16:24:19.027 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:19.035 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:19.035 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:19.091 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:19.092 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:19.200 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:19.225 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:19.324 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.424 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-03-25T16:24:19.749 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:20.002 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:48 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:20.029 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:20.029 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:20.029 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:20.029 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:20.030 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:20.030 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:20.153 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:20.158 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.1 M 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 13 M 2024-03-25T16:24:20.159 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:20.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-03-25T16:24:20.845 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:20.998 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:20.999 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:20.999 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:24:20.999 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:20.999 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.1 M 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 13 M 2024-03-25T16:24:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:27.561 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 482 kB/s | 3.1 MB 00:06 2024-03-25T16:24:27.562 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:27.562 INFO:teuthology.orchestra.run.smithi084.stdout:Total 482 kB/s | 3.1 MB 00:06 2024-03-25T16:24:27.562 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:27.572 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:27.573 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:27.646 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:27.646 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:28.070 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:28.137 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:28.162 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:28.162 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:24:28.162 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:28.162 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:29.316 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:29.317 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:29.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-03-25T16:24:30.137 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:30.286 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:30.286 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:24:30.287 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:30.288 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 171 k 2024-03-25T16:24:30.288 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2024-03-25T16:24:30.288 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:31.055 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 224 kB/s | 171 kB 00:00 2024-03-25T16:24:31.055 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:31.056 INFO:teuthology.orchestra.run.smithi084.stdout:Total 223 kB/s | 171 kB 00:00 2024-03-25T16:24:31.056 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:24:31.065 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:24:31.065 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:24:31.122 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:24:31.122 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:24:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:24:31.292 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 284 kB/s | 3.1 MB 00:11 2024-03-25T16:24:31.293 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:31.294 INFO:teuthology.orchestra.run.smithi157.stdout:Total 284 kB/s | 3.1 MB 00:11 2024-03-25T16:24:31.294 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:31.305 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:31.305 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:31.378 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:31.383 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:31.383 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:31.787 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:31.811 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:31.923 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:31.923 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:24:31.923 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:31.923 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:32.058 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:24:32.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:24:32.880 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:04:09 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:24:33.031 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:24:33.031 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:33.032 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 91 M 2024-03-25T16:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 498 M 2024-03-25T16:24:33.034 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:24:33.271 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-03-25T16:24:33.857 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:34.008 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:34.009 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 171 k 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 494 k 2024-03-25T16:24:34.010 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:24:34.787 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 221 kB/s | 171 kB 00:00 2024-03-25T16:24:34.788 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:34.788 INFO:teuthology.orchestra.run.smithi157.stdout:Total 220 kB/s | 171 kB 00:00 2024-03-25T16:24:34.788 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:24:34.797 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:24:34.797 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:24:34.857 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:24:34.858 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:24:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:24:35.119 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:35.890 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:24:36.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:24:36.699 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:24:36.855 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-25T16:24:36.856 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:24:36.857 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 91 M 2024-03-25T16:24:36.857 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 498 M 2024-03-25T16:24:36.857 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:25:29.190 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 409 kB/s | 22 MB 00:56 2024-03-25T16:25:35.932 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 389 kB/s | 22 MB 00:59 2024-03-25T16:27:19.376 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 429 kB/s | 68 MB 02:42 2024-03-25T16:27:19.377 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:19.377 INFO:teuthology.orchestra.run.smithi157.stdout:Total 571 kB/s | 91 MB 02:42 2024-03-25T16:27:19.377 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:27:19.382 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:27:19.382 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:27:19.733 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 419 kB/s | 68 MB 02:46 2024-03-25T16:27:19.734 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:19.735 INFO:teuthology.orchestra.run.smithi084.stdout:Total 556 kB/s | 91 MB 02:46 2024-03-25T16:27:19.735 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:27:19.739 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:27:19.740 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:27:19.978 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:27:19.978 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:27:20.327 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:27:20.328 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:27:23.079 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:27:23.427 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:27:30.346 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:30.500 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:30.536 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:30.682 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:33.934 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:33.935 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:27:34.034 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:34.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:27:34.209 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:34.209 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:27:34.209 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:27:34.209 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:34.210 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:34.210 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:27:34.210 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:27:34.281 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:27:34.441 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:27:34.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:27:35.013 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:27:35.059 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:07:12 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:27:35.171 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:27:35.171 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:27:35.171 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 416 M 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.7 G 2024-03-25T16:27:35.172 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:27:35.207 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:27:35.208 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 416 M 2024-03-25T16:27:35.209 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.7 G 2024-03-25T16:27:35.209 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:47:53.372 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 350 kB/s | 416 MB 20:18 2024-03-25T16:47:53.389 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:53.389 INFO:teuthology.orchestra.run.smithi084.stdout:Total 350 kB/s | 416 MB 20:18 2024-03-25T16:47:53.389 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:47:53.390 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:47:53.390 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:47:55.843 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:47:55.843 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:48:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:48:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:46.660 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:46.913 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:46.913 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:46.913 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:48:46.914 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:48:46.914 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:46.914 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:48:47.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:48:47.729 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:28:24 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:48:47.886 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:48:47.887 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:47.888 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.4 M 2024-03-25T16:48:47.888 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 6.4 M 2024-03-25T16:48:47.888 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:48:52.097 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 347 kB/s | 1.4 MB 00:04 2024-03-25T16:48:52.098 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:48:52.098 INFO:teuthology.orchestra.run.smithi084.stdout:Total 347 kB/s | 1.4 MB 00:04 2024-03-25T16:48:52.099 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:48:52.102 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:48:52.103 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:48:52.147 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:48:52.147 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:48:52.406 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:48:52.513 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:52.894 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:53.167 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:48:53.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:48:53.903 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:28:30 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:48:54.049 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:48:54.050 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:48:54.051 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 176 M 2024-03-25T16:48:54.051 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 753 M 2024-03-25T16:48:54.051 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:53:02.503 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 279 kB/s | 416 MB 25:27 2024-03-25T16:53:02.503 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:53:02.503 INFO:teuthology.orchestra.run.smithi157.stdout:Total 279 kB/s | 416 MB 25:27 2024-03-25T16:53:02.504 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:53:02.506 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:53:02.507 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:53:04.972 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:53:04.972 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:53:47.115 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:53:47.212 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:55.293 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:53:55.540 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:53:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:53:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:53:55.540 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:53:55.799 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:53:56.379 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:33:24 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:53:56.539 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.4 M 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 6.4 M 2024-03-25T16:53:56.540 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:54:03.123 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 222 kB/s | 1.4 MB 00:06 2024-03-25T16:54:03.124 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:54:03.124 INFO:teuthology.orchestra.run.smithi157.stdout:Total 222 kB/s | 1.4 MB 00:06 2024-03-25T16:54:03.125 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:54:03.130 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:54:03.130 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T16:54:03.175 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T16:54:03.175 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T16:54:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T16:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:04.001 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T16:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T16:54:04.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:54:05.031 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:33:33 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T16:54:05.179 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T16:54:05.180 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:54:05.180 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T16:54:05.180 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:54:05.180 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T16:54:05.180 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 176 M 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 753 M 2024-03-25T16:54:05.181 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T16:59:33.496 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 281 kB/s | 176 MB 10:39 2024-03-25T16:59:33.496 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:33.496 INFO:teuthology.orchestra.run.smithi084.stdout:Total 281 kB/s | 176 MB 10:39 2024-03-25T16:59:33.497 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T16:59:33.499 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T16:59:33.499 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T16:59:34.558 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T16:59:34.558 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T16:59:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T16:59:52.864 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:57.441 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:59:57.712 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T16:59:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:59:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:59:57.713 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T16:59:57.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:59:58.493 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:39:35 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T16:59:58.642 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T16:59:58.642 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 672 M 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 3.1 G 2024-03-25T16:59:58.643 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T16:59:59.024 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 508 kB/s | 176 MB 05:53 2024-03-25T16:59:59.025 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:59.025 INFO:teuthology.orchestra.run.smithi157.stdout:Total 508 kB/s | 176 MB 05:53 2024-03-25T16:59:59.025 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T16:59:59.029 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T16:59:59.029 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:00:00.086 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:00:00.087 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:00:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:00:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:22.964 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:00:23.216 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:00:23.434 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:00:23.997 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:39:51 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:00:24.145 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:00:24.145 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:00:24.145 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:00:24.145 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:00:24.146 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:00:24.147 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 672 M 2024-03-25T17:00:24.147 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 3.1 G 2024-03-25T17:00:24.147 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:23:29.914 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 488 kB/s | 672 MB 23:31 2024-03-25T17:23:29.935 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:23:29.935 INFO:teuthology.orchestra.run.smithi084.stdout:Total 488 kB/s | 672 MB 23:31 2024-03-25T17:23:29.935 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:23:29.936 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:23:29.936 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:23:33.881 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:23:33.882 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:24:44.833 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:24:44.960 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:24:54.565 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:24:54.835 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:24:55.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:24:55.683 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:04:32 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:24:55.779 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:24:55.831 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:24:55.832 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-25T17:24:55.832 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:24:55.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:24:56.465 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:04:33 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:24:56.614 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 20 M 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 103 M 2024-03-25T17:24:56.615 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:26:06.980 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 287 kB/s | 20 MB 01:10 2024-03-25T17:26:06.980 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:06.981 INFO:teuthology.orchestra.run.smithi084.stdout:Total 287 kB/s | 20 MB 01:10 2024-03-25T17:26:06.981 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:26:06.984 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:26:06.984 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:26:07.134 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:26:07.134 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:26:09.369 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:26:09.544 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.444 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.701 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:26:10.701 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:26:10.701 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:10.702 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:26:10.702 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:26:10.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:26:11.455 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:05:48 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:26:11.610 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:26:11.610 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 48 M 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 173 M 2024-03-25T17:26:11.611 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:28:31.362 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 350 kB/s | 48 MB 02:19 2024-03-25T17:28:31.363 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:31.363 INFO:teuthology.orchestra.run.smithi084.stdout:Total 350 kB/s | 48 MB 02:19 2024-03-25T17:28:31.363 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:28:31.367 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:28:31.367 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:28:31.679 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:28:31.680 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:28:36.400 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:28:36.584 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:37.907 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:28:38.162 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:28:38.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:28:38.889 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:08:15 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:28:39.037 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:28:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 99 M 2024-03-25T17:28:39.039 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 449 M 2024-03-25T17:28:39.039 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:31:06.785 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 374 kB/s | 672 MB 30:42 2024-03-25T17:31:06.785 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:06.785 INFO:teuthology.orchestra.run.smithi157.stdout:Total 374 kB/s | 672 MB 30:42 2024-03-25T17:31:06.785 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:31:06.790 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:31:06.790 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:31:10.749 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:31:10.750 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:31:36.752 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 568 kB/s | 99 MB 02:57 2024-03-25T17:31:36.753 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:36.753 INFO:teuthology.orchestra.run.smithi084.stdout:Total 568 kB/s | 99 MB 02:57 2024-03-25T17:31:36.753 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:31:36.757 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:31:36.757 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:31:37.363 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:31:37.363 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:31:47.612 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:31:47.716 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:50.760 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:50.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:31:50.760 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:31:50.760 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:50.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:31:50.761 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:31:50.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:31:51.502 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:11:28 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:31:51.652 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:31:51.652 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 287 M 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.2 G 2024-03-25T17:31:51.653 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:32:23.456 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:32:24.011 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:32.334 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:32:32.591 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:32:32.891 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:32:33.474 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:12:01 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:32:33.576 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:32:33.631 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:32:33.632 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-25T17:32:33.632 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:32:33.721 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:32:34.313 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:12:02 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:32:34.470 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:32:34.471 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:32:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:32:34.471 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:32:34.471 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 20 M 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 103 M 2024-03-25T17:32:34.472 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:33:37.811 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 318 kB/s | 20 MB 01:03 2024-03-25T17:33:37.812 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:37.812 INFO:teuthology.orchestra.run.smithi157.stdout:Total 318 kB/s | 20 MB 01:03 2024-03-25T17:33:37.812 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:33:37.817 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:33:37.817 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:33:37.966 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:33:37.966 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:33:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:33:40.351 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:41.311 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:41.562 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:41.563 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:33:41.563 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:33:41.563 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:41.563 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:33:41.563 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:33:41.740 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:33:42.308 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:13:10 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:33:42.456 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 48 M 2024-03-25T17:33:42.457 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 173 M 2024-03-25T17:33:42.458 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:36:23.658 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 303 kB/s | 48 MB 02:41 2024-03-25T17:36:23.658 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:23.658 INFO:teuthology.orchestra.run.smithi157.stdout:Total 303 kB/s | 48 MB 02:41 2024-03-25T17:36:23.658 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:36:23.661 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:36:23.661 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:36:23.973 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:36:23.973 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:36:28.697 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:36:28.795 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:30.278 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:36:30.533 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:36:30.708 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:36:31.269 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:15:59 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:36:31.417 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:36:31.418 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:36:31.419 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 99 M 2024-03-25T17:36:31.419 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 449 M 2024-03-25T17:36:31.419 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:39:12.994 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 624 kB/s | 99 MB 02:41 2024-03-25T17:39:12.994 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:12.994 INFO:teuthology.orchestra.run.smithi157.stdout:Total 624 kB/s | 99 MB 02:41 2024-03-25T17:39:12.995 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:39:12.997 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:39:12.997 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:39:13.607 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:39:13.607 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:39:23.908 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:39:24.016 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:26.917 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:39:27.179 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:39:27.382 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:39:27.949 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:18:55 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:39:28.097 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:39:28.098 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:39:28.099 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:39:28.099 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 287 M 2024-03-25T17:39:28.099 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.2 G 2024-03-25T17:39:28.099 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:45:14.574 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 366 kB/s | 287 MB 13:22 2024-03-25T17:45:14.574 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:14.574 INFO:teuthology.orchestra.run.smithi084.stdout:Total 366 kB/s | 287 MB 13:22 2024-03-25T17:45:14.575 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:45:14.577 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:45:14.577 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:45:16.292 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:45:16.292 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:45:45.782 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:45:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:53.246 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:45:53.527 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:45:53.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:45:54.300 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:25:31 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:45:54.449 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:45:54.449 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:45:54.449 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:45:54.449 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:45:54.450 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:45:54.451 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.9 M 2024-03-25T17:45:54.451 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 28 M 2024-03-25T17:45:54.451 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:46:20.597 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 233 kB/s | 5.9 MB 00:26 2024-03-25T17:46:20.597 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:20.598 INFO:teuthology.orchestra.run.smithi084.stdout:Total 233 kB/s | 5.9 MB 00:26 2024-03-25T17:46:20.598 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:46:20.602 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:46:20.602 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:46:20.674 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:46:20.674 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:46:21.408 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:46:21.520 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:22.322 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:22.323 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:22.323 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:46:22.323 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:22.323 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:22.323 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:46:22.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:46:23.046 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:26:00 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:46:23.194 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:23.195 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 109 k 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 440 k 2024-03-25T17:46:23.196 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:46:23.957 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 143 kB/s | 109 kB 00:00 2024-03-25T17:46:23.958 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:23.958 INFO:teuthology.orchestra.run.smithi084.stdout:Total 143 kB/s | 109 kB 00:00 2024-03-25T17:46:23.958 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:46:23.962 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:46:23.962 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:46:23.997 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:46:23.997 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:46:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:46:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:46:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:24.839 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:46:25.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:46:25.573 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:26:02 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:46:25.721 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:46:25.722 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:46:25.722 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.4 M 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 25 M 2024-03-25T17:46:25.723 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:46:42.749 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 322 kB/s | 5.4 MB 00:17 2024-03-25T17:46:42.749 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:42.750 INFO:teuthology.orchestra.run.smithi084.stdout:Total 322 kB/s | 5.4 MB 00:17 2024-03-25T17:46:42.750 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:46:42.754 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:46:42.754 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:46:42.823 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:46:42.823 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:46:43.491 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:46:43.605 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:46:44.133 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:46:44.392 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:46:44.393 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:44.393 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:46:44.393 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:44.393 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:44.393 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:46:44.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2-debuginfo 2024-03-25T17:46:45.162 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:26:22 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:46:45.310 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:46:45.311 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:46:45.312 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:46:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:46:45.312 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 44 M 2024-03-25T17:46:45.312 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 253 M 2024-03-25T17:46:45.312 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:49:01.582 INFO:teuthology.orchestra.run.smithi084.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 333 kB/s | 44 MB 02:16 2024-03-25T17:49:01.583 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:01.583 INFO:teuthology.orchestra.run.smithi084.stdout:Total 333 kB/s | 44 MB 02:16 2024-03-25T17:49:01.583 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:49:01.586 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:49:01.586 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:49:01.886 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:49:01.887 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:49:06.956 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:49:07.055 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:09.118 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:09.119 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:49:09.119 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:49:09.119 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:09.119 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:49:09.119 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:49:09.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:49:09.882 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:28:46 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:49:10.034 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:49:10.035 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:49:10.035 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:49:10.035 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:49:10.035 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:49:10.035 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 52 M 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 308 M 2024-03-25T17:49:10.036 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:22.017 INFO:teuthology.orchestra.run.smithi084.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 402 kB/s | 52 MB 02:11 2024-03-25T17:51:22.017 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:22.017 INFO:teuthology.orchestra.run.smithi084.stdout:Total 402 kB/s | 52 MB 02:11 2024-03-25T17:51:22.018 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:22.020 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:22.020 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:22.363 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:22.364 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:28.340 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:30.532 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:30.796 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:30.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:51:31.547 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:08 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:31.697 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:31.699 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 615 k 2024-03-25T17:51:31.699 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.1 M 2024-03-25T17:51:31.699 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:35.225 INFO:teuthology.orchestra.run.smithi084.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 174 kB/s | 615 kB 00:03 2024-03-25T17:51:35.225 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:35.226 INFO:teuthology.orchestra.run.smithi084.stdout:Total 174 kB/s | 615 kB 00:03 2024-03-25T17:51:35.226 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:35.230 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:35.230 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:35.272 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:35.272 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:35.457 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:35.558 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:51:35.987 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:51:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:51:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:36.243 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:36.243 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:36.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:51:36.979 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:13 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:37.127 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:37.129 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:37.129 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:37.129 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 771 k 2024-03-25T17:51:37.129 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.7 M 2024-03-25T17:51:37.129 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:40.163 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 254 kB/s | 771 kB 00:03 2024-03-25T17:51:40.163 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:40.164 INFO:teuthology.orchestra.run.smithi084.stdout:Total 254 kB/s | 771 kB 00:03 2024-03-25T17:51:40.164 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:40.168 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:40.168 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:40.208 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:40.209 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:40.394 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:40.502 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:51:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:41.142 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:41.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:51:41.907 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:18 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:42.055 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:42.056 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:42.056 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:42.056 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.1 M 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 4.3 M 2024-03-25T17:51:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:45.165 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 370 kB/s | 1.1 MB 00:03 2024-03-25T17:51:45.166 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:45.166 INFO:teuthology.orchestra.run.smithi084.stdout:Total 370 kB/s | 1.1 MB 00:03 2024-03-25T17:51:45.166 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:45.170 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:45.170 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:45.213 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:45.214 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:45.448 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:45.563 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:45.967 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:46.226 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:46.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:51:47.029 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:24 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:47.180 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:47.181 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:47.181 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 337 k 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.0 M 2024-03-25T17:51:47.182 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 246 kB/s | 337 kB 00:01 2024-03-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi084.stdout:Total 246 kB/s | 337 kB 00:01 2024-03-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:49.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:51:50.271 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:27 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:50.420 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:50.421 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:50.421 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 509 k 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.5 M 2024-03-25T17:51:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:51:54.187 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 135 kB/s | 509 kB 00:03 2024-03-25T17:51:54.187 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:54.188 INFO:teuthology.orchestra.run.smithi084.stdout:Total 135 kB/s | 509 kB 00:03 2024-03-25T17:51:54.188 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:51:54.192 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:51:54.192 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:51:54.231 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:51:54.232 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:51:54.399 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:51:54.494 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:54.876 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:55.133 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:51:55.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:51:55.941 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:31:32 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:51:56.092 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:51:56.092 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:56.092 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:51:56.092 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:56.092 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:51:56.093 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 52 M 2024-03-25T17:51:56.094 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 298 M 2024-03-25T17:51:56.094 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:53:43.095 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 493 kB/s | 52 MB 01:46 2024-03-25T17:53:43.095 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:43.095 INFO:teuthology.orchestra.run.smithi084.stdout:Total 493 kB/s | 52 MB 01:47 2024-03-25T17:53:43.096 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:53:43.098 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:53:43.098 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:53:43.434 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:53:43.435 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:53:49.356 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:53:49.542 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:53:51.568 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:53:51.829 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:53:51.830 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:51.830 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:53:51.830 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:51.830 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:51.830 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:53:52.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:53:52.591 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 1:33:29 ago on Mon 25 Mar 2024 04:20:23 PM UTC. 2024-03-25T17:53:52.739 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-25T17:53:52.740 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-25T17:53:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:52.741 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.3 M 2024-03-25T17:53:52.741 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 5.5 M 2024-03-25T17:53:52.741 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-25T17:53:58.436 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 225 kB/s | 1.3 MB 00:05 2024-03-25T17:53:58.436 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:58.437 INFO:teuthology.orchestra.run.smithi084.stdout:Total 225 kB/s | 1.3 MB 00:05 2024-03-25T17:53:58.437 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-25T17:53:58.441 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-25T17:53:58.441 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-25T17:53:58.485 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-25T17:53:58.486 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-25T17:53:58.723 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-25T17:53:58.826 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:59.250 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T17:53:59.520 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-25T17:53:59.706 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:57.940 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 316 kB/s | 287 MB 15:29 2024-03-25T17:54:57.941 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:57.941 INFO:teuthology.orchestra.run.smithi157.stdout:Total 316 kB/s | 287 MB 15:29 2024-03-25T17:54:57.941 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:54:57.944 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:54:57.944 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:54:59.652 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:54:59.652 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:55:29.043 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:55:29.637 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:33.277 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:33.537 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:33.537 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:33.537 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:55:33.538 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:33.538 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:33.538 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:55:33.765 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:55:34.338 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:35:02 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:55:34.487 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:34.488 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:55:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:34.489 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 5.9 M 2024-03-25T17:55:34.489 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 28 M 2024-03-25T17:55:34.489 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:55:53.277 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 324 kB/s | 5.9 MB 00:18 2024-03-25T17:55:53.278 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:53.278 INFO:teuthology.orchestra.run.smithi157.stdout:Total 324 kB/s | 5.9 MB 00:18 2024-03-25T17:55:53.278 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:55:53.282 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:55:53.282 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:55:53.355 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:55:53.356 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:55:54.185 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:55.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:55.029 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:55.029 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:55:55.029 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:55.029 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:55.029 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:55:55.236 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:55:55.805 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:35:23 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:55:55.954 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 109 k 2024-03-25T17:55:55.955 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 440 k 2024-03-25T17:55:55.956 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:55:56.772 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 133 kB/s | 109 kB 00:00 2024-03-25T17:55:56.773 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:56.773 INFO:teuthology.orchestra.run.smithi157.stdout:Total 133 kB/s | 109 kB 00:00 2024-03-25T17:55:56.773 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:55:56.776 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:55:56.776 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:55:56.814 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:55:56.814 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:55:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:55:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:57.421 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:57.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:57.676 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:57.676 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:55:57.676 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:57.676 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:57.676 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:55:57.868 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:55:58.435 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:35:26 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 5.4 M 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 25 M 2024-03-25T17:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:56:16.589 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 304 kB/s | 5.4 MB 00:18 2024-03-25T17:56:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Total 304 kB/s | 5.4 MB 00:18 2024-03-25T17:56:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:56:16.594 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:56:16.594 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:56:16.661 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:56:16.662 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:56:17.325 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:56:17.421 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:17.948 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:56:18.207 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:56:18.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2-debuginfo 2024-03-25T17:56:18.989 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:35:46 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:56:19.137 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:56:19.138 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:56:19.138 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:56:19.138 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 44 M 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 253 M 2024-03-25T17:56:19.139 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T17:58:24.200 INFO:teuthology.orchestra.run.smithi157.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 363 kB/s | 44 MB 02:05 2024-03-25T17:58:24.200 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:24.200 INFO:teuthology.orchestra.run.smithi157.stdout:Total 363 kB/s | 44 MB 02:05 2024-03-25T17:58:24.200 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T17:58:24.203 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T17:58:24.203 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T17:58:24.501 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T17:58:24.502 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T17:58:29.550 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T17:58:29.647 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:31.491 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:58:31.745 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T17:58:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:58:31.745 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T17:58:31.938 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:58:32.507 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:38:00 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T17:58:32.658 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T17:58:32.659 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:58:32.659 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T17:58:32.659 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:58:32.659 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T17:58:32.659 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 52 M 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 308 M 2024-03-25T17:58:32.660 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:00:57.923 INFO:teuthology.orchestra.run.smithi157.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 365 kB/s | 52 MB 02:25 2024-03-25T18:00:57.924 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:57.924 INFO:teuthology.orchestra.run.smithi157.stdout:Total 365 kB/s | 52 MB 02:25 2024-03-25T18:00:57.924 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:00:57.926 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:00:57.926 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:00:58.270 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:00:58.270 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:04.496 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:06.654 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:06.908 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:06.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:06.909 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:06.909 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:06.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:06.909 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:07.102 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:01:07.661 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:35 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:07.809 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:07.810 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:07.810 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:07.810 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:07.810 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 615 k 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.1 M 2024-03-25T18:01:07.811 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:01:11.124 INFO:teuthology.orchestra.run.smithi157.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 186 kB/s | 615 kB 00:03 2024-03-25T18:01:11.124 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:11.124 INFO:teuthology.orchestra.run.smithi157.stdout:Total 186 kB/s | 615 kB 00:03 2024-03-25T18:01:11.125 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:01:11.129 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:01:11.129 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:01:11.170 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:01:11.170 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:11.340 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:11.438 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:11.865 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:12.126 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:12.127 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:12.127 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:12.127 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:12.127 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:12.127 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:12.331 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:01:12.899 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:40 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:13.050 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:13.051 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:13.052 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:13.052 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 771 k 2024-03-25T18:01:13.052 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.7 M 2024-03-25T18:01:13.052 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:01:15.454 INFO:teuthology.orchestra.run.smithi157.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 321 kB/s | 771 kB 00:02 2024-03-25T18:01:15.454 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:15.455 INFO:teuthology.orchestra.run.smithi157.stdout:Total 321 kB/s | 771 kB 00:02 2024-03-25T18:01:15.455 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:01:15.459 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:01:15.459 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:01:15.501 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:01:15.501 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:15.689 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:15.786 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:16.191 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:16.449 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:16.450 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:16.667 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:01:17.229 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:45 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:17.378 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:17.379 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:17.380 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.1 M 2024-03-25T18:01:17.380 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 4.3 M 2024-03-25T18:01:17.380 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:01:21.673 INFO:teuthology.orchestra.run.smithi157.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 268 kB/s | 1.1 MB 00:04 2024-03-25T18:01:21.673 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:21.674 INFO:teuthology.orchestra.run.smithi157.stdout:Total 268 kB/s | 1.1 MB 00:04 2024-03-25T18:01:21.674 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:01:21.678 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:01:21.678 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:01:21.722 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:01:21.722 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:21.943 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:22.431 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:22.690 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:22.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:01:23.479 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:51 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:23.629 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:23.630 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 337 k 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.0 M 2024-03-25T18:01:23.631 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:01:25.959 INFO:teuthology.orchestra.run.smithi157.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 145 kB/s | 337 kB 00:02 2024-03-25T18:01:25.960 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:25.960 INFO:teuthology.orchestra.run.smithi157.stdout:Total 145 kB/s | 337 kB 00:02 2024-03-25T18:01:25.960 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:01:25.964 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:01:25.964 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:01:26.004 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:01:26.004 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:26.145 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:26.243 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:26.881 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:26.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:26.881 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:26.881 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:26.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:26.882 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:27.090 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:01:27.654 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:55 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:27.803 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:27.803 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 509 k 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.5 M 2024-03-25T18:01:27.804 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:01:29.920 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 241 kB/s | 509 kB 00:02 2024-03-25T18:01:29.921 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:29.922 INFO:teuthology.orchestra.run.smithi157.stdout:Total 240 kB/s | 509 kB 00:02 2024-03-25T18:01:29.922 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:01:29.926 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:01:29.926 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:01:29.966 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:01:29.966 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:01:30.125 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:01:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:30.923 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:30.923 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:30.924 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:01:30.924 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:30.924 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:30.924 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:01:31.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:01:31.697 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:40:59 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:01:31.854 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:01:31.855 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:01:31.856 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:01:31.856 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:01:31.856 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 52 M 2024-03-25T18:01:31.856 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 298 M 2024-03-25T18:01:31.856 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:04:14.294 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 325 kB/s | 52 MB 02:42 2024-03-25T18:04:14.294 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:14.294 INFO:teuthology.orchestra.run.smithi157.stdout:Total 325 kB/s | 52 MB 02:42 2024-03-25T18:04:14.295 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:04:14.296 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:04:14.297 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:04:14.629 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:04:14.629 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:04:20.509 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:04:20.823 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:22.851 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:23.109 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:04:23.303 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:04:23.881 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 1:43:51 ago on Mon 25 Mar 2024 04:20:32 PM UTC. 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:04:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:24.032 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-25T18:04:24.032 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-25T18:04:24.032 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-25T18:04:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:24.032 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.3 M 2024-03-25T18:04:24.033 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 5.5 M 2024-03-25T18:04:24.033 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-25T18:04:27.544 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 365 kB/s | 1.3 MB 00:03 2024-03-25T18:04:27.545 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:27.545 INFO:teuthology.orchestra.run.smithi157.stdout:Total 365 kB/s | 1.3 MB 00:03 2024-03-25T18:04:27.545 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-25T18:04:27.548 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-25T18:04:27.549 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-25T18:04:27.593 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-25T18:04:27.594 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-25T18:04:27.834 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-25T18:04:27.934 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:28.327 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:28.635 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:28.635 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:28.635 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-25T18:04:28.635 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.635 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:28.636 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-25T18:04:28.865 DEBUG:teuthology.parallel:result is None 2024-03-25T18:04:28.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:04:29.014 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:04:29.039 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:04:29.040 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:04:29.040 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:04:29.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:04:29.177 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:04:29.198 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:04:29.198 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:04:29.198 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:04:29.200 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:04:29.200 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:29.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:04:29.233 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:29.234 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:04:29.266 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:04:29.266 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:29.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:04:29.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:04:29.375 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:29.376 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:04:29.402 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:04:29.469 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:04:29.469 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:29.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:04:29.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:04:29.567 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:29.567 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:04:29.594 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:04:29.661 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:04:29.661 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:29.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:04:29.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:04:29.757 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:29.757 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:04:29.784 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:04:29.851 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:04:29.926 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:04:29.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2024-03-25T18:04:29.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-03-25T18:04:29.956 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:04:29.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:04:29.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:04:30.025 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:04:30.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:04:30.067 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:04:30.130 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:04:30.131 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:04:30.131 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621255', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3t1f7dXKRia+bTNuo77Ot0iB2dDzXvpcv7bpKqPxqen4vit2GWi0a+v/UoPo9CIZ+qZGjp4U2gYFz1A6NfDAI=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAW6YOCv5MC8vYhC9glGx5t7q8Pdb+sqj/Fviye5coELMdd8qf1QHkf4B+qZAVgJRcyOxlZUIQBMF9DZM5dI6a4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:04:30.131 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:04:30.136 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:04:30.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:04:30.192 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:04:30.219 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:30.219 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:04:30.273 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:04:30.273 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-03-25T18:04:30.329 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:04:30.329 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-25 16:24:28.419123476 +0000 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-25 16:19:28.855938641 +0000 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-25 16:19:28.855938641 +0000 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-25 16:19:28.855938641 +0000 2024-03-25T18:04:30.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:04:30.395 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-25T18:04:30.395 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-25T18:04:30.395 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000258696 s, 2.0 MB/s 2024-03-25T18:04:30.396 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:04:30.452 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-25 16:24:28.420123458 +0000 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-25 16:19:29.194932798 +0000 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-25 16:19:29.194932798 +0000 2024-03-25T18:04:30.507 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-25 16:19:29.194932798 +0000 2024-03-25T18:04:30.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:04:30.573 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-25T18:04:30.573 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-25T18:04:30.573 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000206447 s, 2.5 MB/s 2024-03-25T18:04:30.575 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:04:30.630 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-25 16:24:28.420123458 +0000 2024-03-25T18:04:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-25 16:19:29.531926990 +0000 2024-03-25T18:04:30.686 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-25 16:19:29.531926990 +0000 2024-03-25T18:04:30.686 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-25 16:19:29.531926990 +0000 2024-03-25T18:04:30.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:04:30.751 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-25T18:04:30.751 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-25T18:04:30.751 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.00020338 s, 2.5 MB/s 2024-03-25T18:04:30.752 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:04:30.808 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-03-25T18:04:30.862 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:04:30.862 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:30.862 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:04:30.862 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:30.863 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:30.863 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-25 16:24:28.421123441 +0000 2024-03-25T18:04:30.863 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-25 16:19:29.889920820 +0000 2024-03-25T18:04:30.863 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-25 16:19:29.889920820 +0000 2024-03-25T18:04:30.863 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-25 16:19:29.889920820 +0000 2024-03-25T18:04:30.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:04:30.928 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-25T18:04:30.928 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-25T18:04:30.928 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000198762 s, 2.6 MB/s 2024-03-25T18:04:30.929 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:04:30.985 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:30.985 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:30.985 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:04:31.001 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:04:31.001 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:31.057 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-25 16:24:32.176643726 +0000 2024-03-25T18:04:31.058 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-25 16:19:22.853285028 +0000 2024-03-25T18:04:31.058 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-25 16:19:22.853285028 +0000 2024-03-25T18:04:31.058 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-25 16:19:22.853285028 +0000 2024-03-25T18:04:31.058 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:04:31.124 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-25T18:04:31.124 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-25T18:04:31.124 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000104456 s, 4.9 MB/s 2024-03-25T18:04:31.125 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:04:31.182 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-03-25T18:04:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:04:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:31.237 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-25 16:24:32.176643726 +0000 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-25 16:19:23.194278302 +0000 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-25 16:19:23.194278302 +0000 2024-03-25T18:04:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-25 16:19:23.194278302 +0000 2024-03-25T18:04:31.238 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:04:31.303 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-25T18:04:31.303 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-25T18:04:31.303 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000101624 s, 5.0 MB/s 2024-03-25T18:04:31.304 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:04:31.360 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-25 16:24:32.177643707 +0000 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-25 16:19:23.533271615 +0000 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-25 16:19:23.533271615 +0000 2024-03-25T18:04:31.415 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-25 16:19:23.533271615 +0000 2024-03-25T18:04:31.415 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:04:31.480 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-25T18:04:31.480 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-25T18:04:31.480 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000103538 s, 4.9 MB/s 2024-03-25T18:04:31.482 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:04:31.538 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:04:31.592 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-25 16:24:32.177643707 +0000 2024-03-25T18:04:31.593 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-25 16:19:23.876264850 +0000 2024-03-25T18:04:31.593 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-25 16:19:23.876264850 +0000 2024-03-25T18:04:31.593 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-25 16:19:23.876264850 +0000 2024-03-25T18:04:31.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:04:31.658 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-25T18:04:31.658 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-25T18:04:31.658 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 7.4109e-05 s, 6.9 MB/s 2024-03-25T18:04:31.659 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:04:31.715 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:31.715 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:04:31.715 INFO:tasks.ceph:Generating config... 2024-03-25T18:04:31.716 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:04:31.716 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:04:31.716 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:04:31.716 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:04:31.716 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:04:31.716 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:04:31.716 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:04:31.717 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:04:31.717 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:04:31.717 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:04:31.717 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:04:31.718 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:04:31.718 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:04:31.719 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:04:31.720 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:04:31.720 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:04:31.720 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:04:31.720 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:04:31.720 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:04:31.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:04:31.775 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:04:31.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:04:31.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:04:31.891 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.84:6789]'), ('mon.c', '[v1:172.21.15.84:6790]'), ('mon.b', '[v1:172.21.15.157:6789]')] 2024-03-25T18:04:31.891 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.157:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:04:31.892 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:31.892 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:04:31.946 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.84:6789]' --addv c '[v1:172.21.15.84:6790]' --addv b '[v1:172.21.15.157:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:32.030 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: generated fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 0 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:min_mon_release 17 (quincy) 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:election_strategy: 3 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:04:32.031 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:04:32.032 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:04:32.047 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 790a4795-8265-4274-9bee-7eca8ec16c87... 2024-03-25T18:04:32.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:04:32.089 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.136 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:04:32.137 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:04:32.138 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.157:6789]" 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon client directed command retry = 5 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: mon election default strategy = 3 2024-03-25T18:04:32.139 INFO:teuthology.orchestra.run.smithi084.stdout: ms bind msgr2 = False 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: ms inject socket failures = 5000 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: ms type = async 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd heartbeat grace = 80 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2024-03-25T18:04:32.140 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2024-03-25T18:04:32.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bdev async discard = True 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bdev enable discard = True 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore allocator = stupid 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore block size = 96636764160 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore fsck on mount = True 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore zero block detection = True 2024-03-25T18:04:32.142 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluefs = 1 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluestore = 1 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: debug rocksdb = 10 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd full ratio = 0.9 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: osd class default list = * 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: osd class load list = * 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: osd client watch timeout = 120 2024-03-25T18:04:32.143 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify cached snaps = True 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify missing on start = True 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd fast shutdown = False 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd max markdown count = 1000 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock override recovery settings = True 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd objectstore = bluestore 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue = debug_random 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue cut off = debug_random 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: osd shutdown pgref assert = True 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.144 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-25T18:04:32.145 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.146 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon min osdmap epochs = 50 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd crush smoke test = False 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune interval = 2 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune min = 15 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:04:32.147 INFO:teuthology.orchestra.run.smithi084.stdout: mon scrub interval = 300 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on pool no app = False 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: paxos service trim min = 10 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: debug objecter = 20 2024-03-25T18:04:32.148 INFO:teuthology.orchestra.run.smithi084.stdout: debug rados = 20 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.c] 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.b] 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-03-25T18:04:32.149 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-03-25T18:04:32.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-03-25T18:04:32.151 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-03-25T18:04:32.152 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.157:6789]" 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: mon client directed command retry = 5 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 3 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: ms bind msgr2 = False 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: ms inject socket failures = 5000 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: osd heartbeat grace = 80 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.153 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-03-25T18:04:32.154 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: bdev async discard = True 2024-03-25T18:04:32.155 INFO:teuthology.orchestra.run.smithi157.stdout: bdev enable discard = True 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore allocator = stupid 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 1 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 1 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:04:32.156 INFO:teuthology.orchestra.run.smithi157.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd class default list = * 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd class load list = * 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd client watch timeout = 120 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd fast shutdown = False 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2024-03-25T18:04:32.157 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-25T18:04:32.158 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd crush smoke test = False 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on pool no app = False 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2024-03-25T18:04:32.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: debug objecter = 20 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: debug rados = 20 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-03-25T18:04:32.161 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-03-25T18:04:32.163 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:04:32.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:04:32.217 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:04:32.218 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:32.218 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:04:32.232 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:32.233 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:04:32.287 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:04:32.287 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:32.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:04:32.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:04:32.367 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:04:32.367 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:32.421 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:04:32.422 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:32.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:04:32.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:04:32.460 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:04:32.461 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:32.515 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:04:32.515 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:04:32.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:04:32.569 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:04:32.572 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:04:32.635 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:04:32.637 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:04:32.637 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:04:32.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:04:32.679 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:04:32.695 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:04:32.737 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:04:32.753 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:04:32.753 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:04:32.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:04:32.780 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:32.780 INFO:tasks.ceph:role: osd.0 2024-03-25T18:04:32.780 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:04:32.780 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:04:32.854 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:32.854 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:33.359 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-25T18:04:33.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-25T18:04:33.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:04:33.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:04:33.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:04:33.590 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:33.590 INFO:tasks.ceph:role: osd.1 2024-03-25T18:04:33.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:04:33.590 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:33.665 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:34.130 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-25T18:04:34.132 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-25T18:04:34.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:04:34.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:04:34.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:04:34.354 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:34.354 INFO:tasks.ceph:role: osd.2 2024-03-25T18:04:34.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:04:34.354 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:34.429 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:34.429 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:34.429 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:34.957 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-25T18:04:34.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-25T18:04:34.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:04:35.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:04:35.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:04:35.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:35.183 INFO:tasks.ceph:role: osd.3 2024-03-25T18:04:35.183 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:04:35.184 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:35.259 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:35.260 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:35.743 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-25T18:04:35.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-25T18:04:35.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:04:35.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:04:35.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:36.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:04:36.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:04:36.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:04:36.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:04:36.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:04:36.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:36.013+0000 7f2a73373640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:04:38.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:04:38.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:38.977 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.977+0000 7f01ab193640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:04:38.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.978+0000 7f01ab193640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:04:38.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.978+0000 7f01ab193640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:04:38.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.978+0000 7f01ab193640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:04:38.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.978+0000 7f01ab193640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:04:38.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:38.978+0000 7f01ab193640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:04:41.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:04:41.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:41.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.947+0000 7f8d8d62a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:04:41.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.947+0000 7f8d8d62a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:04:41.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.947+0000 7f8d8d62a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:04:41.948 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.947+0000 7f8d8d62a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:04:41.948 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.947+0000 7f8d8d62a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:04:41.948 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:41.948+0000 7f8d8d62a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:04:44.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:04:44.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:44.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.911+0000 7faaa24c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:04:44.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.911+0000 7faaa24c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:04:44.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.911+0000 7faaa24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:04:44.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.911+0000 7faaa24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:04:44.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.911+0000 7faaa24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:04:44.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:04:44.912+0000 7faaa24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:04:47.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:04:47.783 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:04:47.815 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:47.815 INFO:tasks.ceph:role: osd.4 2024-03-25T18:04:47.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:04:47.815 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:47.886 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:47.919 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-25T18:04:48.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-25T18:04:48.122 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:04:48.198 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:04:48.272 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:04:48.344 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:48.344 INFO:tasks.ceph:role: osd.5 2024-03-25T18:04:48.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:04:48.344 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:48.411 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:48.412 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:48.412 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:48.412 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:48.444 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-25T18:04:48.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-25T18:04:48.446 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:04:48.522 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:04:48.594 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:04:48.661 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:48.661 INFO:tasks.ceph:role: osd.6 2024-03-25T18:04:48.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:04:48.662 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:04:48.728 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:48.729 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:48.761 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-25T18:04:48.763 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-25T18:04:48.764 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:04:48.846 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:04:48.917 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:04:48.984 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:04:48.984 INFO:tasks.ceph:role: osd.7 2024-03-25T18:04:48.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:04:48.984 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:04:49.051 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:04:49.083 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-25T18:04:49.085 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-25T18:04:49.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:04:49.156 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:04:49.227 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:04:49.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:49.323+0000 7fb9ee4d5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:04:52.162 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:04:52.194 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:52.288 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.287+0000 7f0354e14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:04:52.288 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.287+0000 7f0354e14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:04:52.288 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.287+0000 7f0354e14640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:04:52.289 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.287+0000 7f0354e14640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:04:52.289 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.287+0000 7f0354e14640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:04:52.289 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:52.288+0000 7f0354e14640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:04:55.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:04:55.161 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:55.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.254+0000 7fd9719dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:04:55.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.255+0000 7fd9719dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:04:55.256 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.255+0000 7fd9719dd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:04:55.256 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.255+0000 7fd9719dd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:04:55.256 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.255+0000 7fd9719dd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:04:55.256 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:55.255+0000 7fd9719dd640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:04:58.091 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:04:58.122 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:04:58.214 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:04:58.214 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:04:58.215 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:04:58.215 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:04:58.215 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:04:58.215 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:04:58.213+0000 7f9083e12640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:05:01.050 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:05:01.080 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:05:01.080 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:05:01.112 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:05:01.183 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:05:01.254 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:05:01.324 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:05:01.394 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.394 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:05:01.425 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:05:01.495 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.495 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:05:01.564 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.564 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:05:01.633 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.633 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:05:01.701 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:01.701 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:05:01.719 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:01.719 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:05:01.757 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:05:01.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:05:01.761 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:05:01.800 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.y] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCwvAFmWND8IRAAjN3zbDJMRoMk1LTfrtpfWQ== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC0vAFmJE7OABAAEFH1vFzDmjGynCbbP/QBjw== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC2vAFmYAxOOhAA1lMtDDYK0qjHrB3bP7N8Sg== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC5vAFmbNN9OBAApoicsyJMX17WRWrTOYjlLQ== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC8vAFmg/dlNhAAVFEBoR9GnQOcQGZO0QzY/Q== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCwvAFmFgniJRAABV24Z64LMgLfzXnSmtxFAQ== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDBvAFmo0FSExAAW4vuSbRrj1e4JtoEatPCbg== 2024-03-25T18:05:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDEvAFm2+EzERAA6TFyu5Nblgs1L7Jqd6xU9Q== 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDHvAFmlTZADxAAswClQvXzyv2cVnte4J7iqw== 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDKvAFmD6fGDBAALjFzHFwEjF1eA64oEjsyZw== 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCwvAFmEiCSKBAAuSjh1prryH9ZqV6vvnsR/w== 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:[client.1] 2024-03-25T18:05:01.802 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCwvAFmT7L7KxAAttzxhObC3tYu0Of2X2l1cA== 2024-03-25T18:05:01.823 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-03-25T18:05:01.823 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCwvAFmWND8IRAAjN3zbDJMRoMk1LTfrtpfWQ== 2024-03-25T18:05:01.823 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-03-25T18:05:01.823 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC0vAFmJE7OABAAEFH1vFzDmjGynCbbP/QBjw== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC2vAFmYAxOOhAA1lMtDDYK0qjHrB3bP7N8Sg== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC5vAFmbNN9OBAApoicsyJMX17WRWrTOYjlLQ== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC8vAFmg/dlNhAAVFEBoR9GnQOcQGZO0QzY/Q== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCwvAFmFgniJRAABV24Z64LMgLfzXnSmtxFAQ== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDBvAFmo0FSExAAW4vuSbRrj1e4JtoEatPCbg== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDEvAFm2+EzERAA6TFyu5Nblgs1L7Jqd6xU9Q== 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-03-25T18:05:01.824 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDHvAFmlTZADxAAswClQvXzyv2cVnte4J7iqw== 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDKvAFmD6fGDBAALjFzHFwEjF1eA64oEjsyZw== 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCwvAFmEiCSKBAAuSjh1prryH9ZqV6vvnsR/w== 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2024-03-25T18:05:01.825 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCwvAFmT7L7KxAAttzxhObC3tYu0Of2X2l1cA== 2024-03-25T18:05:01.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:05:01.846 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:05:01.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:01.921 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:01.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:01.975 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.029 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.083 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:05:02.137 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:05:02.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.191 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.245 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.299 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.353 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:05:02.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:05:02.407 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:05:02.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:05:02.463 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:05:02.518 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:05:02.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:05:02.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:05:02.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:05:02.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:05:02.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:05:02.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:05:02.856 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:05:02.885 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:05:02.983 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:05:03.010 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:05:03.012 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:05:03.065 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:05:03.065 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:05:03.065 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:05:03.065 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:05:03.068 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:05:03.068 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:05:03.068 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:05:03.068 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:05:03.070 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:05:03.070 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:05:03.071 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:05:03.071 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:05:03.107 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:05:03.107 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:05:03.108 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:05:03.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:05:03.110 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:05:03.110 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:05:03.110 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:05:03.112 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:05:03.112 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:05:03.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.213+0000 7ffa23115640 10 monclient: get_monmap_and_config 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.213+0000 7ffa23115640 10 monclient: build_initial_monmap 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.213+0000 7ffa23115640 10 monclient: monmap: 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:05:03.214609+0000 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:05:03.214609+0000 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:03.214 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.213+0000 7ffa23115640 1 Processor -- start 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 1 -- start start 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient: init 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a9920 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0b45a0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c146450 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c06b100 con 0x7ffa1c0b45a0 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b2940 con 0x7ffa1c0a9920 2024-03-25T18:05:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c13fec0 con 0x7ffa1c146450 2024-03-25T18:05:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa23115640 10 monclient(hunting): authenticate will time out at 6947.513551s 2024-03-25T18:05:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa1c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa1c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.214+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa1c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:03.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.415+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa1c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:03.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.415+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa1c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:03.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.415+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa1c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:03.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.816+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa1c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:03.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.816+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa1c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:03.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:03.816+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa1c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:04.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:04.617+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa1c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:04.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:04.617+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa1c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:04.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:04.617+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa1c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:06.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): tick 2024-03-25T18:05:06.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:05:06.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa1c14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa1c13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa1c145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a9920 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0b45a0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c146450 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c06b100 con 0x7ffa1c0b45a0 2024-03-25T18:05:06.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b2940 con 0x7ffa1c0a9920 2024-03-25T18:05:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c13fec0 con 0x7ffa1c146450 2024-03-25T18:05:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.214+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa000056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:06.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.415+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:06.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.415+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:06.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.415+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa000056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:06.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.816+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:06.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.816+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:06.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:06.816+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa000056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:07.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:07.617+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:07.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:07.617+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:07.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:07.617+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa000056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:09.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): tick 2024-03-25T18:05:09.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:05:09.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:05:09.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:05:09.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa000056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a9920 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c146450 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.214+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0b45a0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa00010e20 con 0x7ffa1c0b45a0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c06b100 con 0x7ffa1c0a9920 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b2940 con 0x7ffa1c146450 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa1a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.215+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:09.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.416+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:09.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.416+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:09.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.416+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:09.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.817+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:09.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.817+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:09.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:09.817+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:10.071 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:00:06.855 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:10.347 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:07.167 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:10.347 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:07.167 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:10.377 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:00:07.161 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:10.618 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:10.618+0000 7ffa216b2640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-25T18:05:10.619 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:10.618+0000 7ffa1bfff640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6790/0 2024-03-25T18:05:10.619 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:10.618+0000 7ffa20eb1640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-25T18:05:10.667 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:07.487 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:10.668 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:07.488 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): tick 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:05:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0b45a0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c146450 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a9920 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c13fec0 con 0x7ffa1c0b45a0 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa00010e20 con 0x7ffa1c0a9920 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c06b100 con 0x7ffa1c146450 2024-03-25T18:05:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.215+0000 7ffa1a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:12.260 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.259+0000 7ffa216b2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 0x7ffa0000fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:48176/0 (socket says 172.21.15.84:48176) 2024-03-25T18:05:12.260 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:12.259+0000 7ffa216b2640 1 -- 172.21.15.84:0/4249115295 learned_addr learned my addr 172.21.15.84:0/4249115295 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): tick 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6790/0 conn(0x7ffa1c146450 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.157:6789/0 conn(0x7ffa1c0a9920 legacy=0x7ffa0000fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6789/0 conn(0x7ffa1c0b45a0 legacy=0x7ffa00005680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa00008f70 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa000120a0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa00015c10 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b2940 con 0x7ffa00015c10 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0000ff60 con 0x7ffa00008f70 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa00010190 con 0x7ffa000120a0 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:15.216+0000 7ffa1a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:18.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): tick 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6790/0 conn(0x7ffa000120a0 legacy=0x7ffa00014490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.157:6789/0 conn(0x7ffa00008f70 legacy=0x7ffa000105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6789/0 conn(0x7ffa00015c10 legacy=0x7ffa00016010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa000056b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa00008f70 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:18.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa000120a0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:18.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000103e0 con 0x7ffa000056b0 2024-03-25T18:05:18.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000154a0 con 0x7ffa00008f70 2024-03-25T18:05:18.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa00015630 con 0x7ffa000120a0 2024-03-25T18:05:18.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:18.216+0000 7ffa1a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.654+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792030323 0 0) 0x7ffa000154a0 con 0x7ffa00008f70 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.654+0000 7ffa1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.654+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.654+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.654+0000 7ffa1b7fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:05:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.655+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0c003520 con 0x7ffa00008f70 2024-03-25T18:05:20.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404120074 0 0) 0x7ffa000103e0 con 0x7ffa000056b0 2024-03-25T18:05:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:05:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.666+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa000154a0 con 0x7ffa000056b0 2024-03-25T18:05:20.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183609605 0 0) 0x7ffa00015630 con 0x7ffa000120a0 2024-03-25T18:05:20.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:20.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:20.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:20.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:05:20.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.672+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa000103e0 con 0x7ffa000120a0 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.723+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460766934 0 0) 0x7ffa000103e0 con 0x7ffa000120a0 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.723+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.157:6789/0 conn(0x7ffa00008f70 legacy=0x7ffa000105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6789/0 conn(0x7ffa000056b0 legacy=0x7ffa0000c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa000157c0 con 0x7ffa000120a0 2024-03-25T18:05:20.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffa14004480 con 0x7ffa000120a0 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa23115640 5 monclient: authenticate success, global_id 4103 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa23115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa1b7fe640 10 monclient: dump: 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:20.726 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:20.726 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:20.726 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:20.726 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:20.726 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.724+0000 7ffa23115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa140045e0 con 0x7ffa000120a0 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 1 -- 172.21.15.84:0/4249115295 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffa14004a30 con 0x7ffa000120a0 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 4 set_mon_vals no callback set 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:20.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa1b7fe640 10 monclient: dump: 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:20.767 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 10 monclient: shutdown 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 1 -- 172.21.15.84:0/4249115295 >> v1:172.21.15.84:6790/0 conn(0x7ffa000120a0 legacy=0x7ffa000095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 1 -- 172.21.15.84:0/4249115295 shutdown_connections 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.766+0000 7ffa23115640 1 -- 172.21.15.84:0/4249115295 >> 172.21.15.84:0/4249115295 conn(0x7ffa1c0b7f40 msgr2=0x7ffa1c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 1 -- 172.21.15.84:0/4249115295 shutdown_connections 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 1 -- 172.21.15.84:0/4249115295 wait complete. 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 10 monclient: build_initial_monmap 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 10 monclient: monmap: 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:20.770 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 1 librados: starting msgr at 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.767+0000 7ffa23115640 1 librados: starting objecter 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 Processor -- start 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 -- start start 2024-03-25T18:05:20.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 librados: setting wanted keys 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 librados: calling monclient init 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient: init 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-a con 0x7ffa000056b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-b con 0x7ffa00008f70 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient(hunting): picked mon.noname-c con 0x7ffa000120a0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c1d7750 con 0x7ffa000056b0 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c143330 con 0x7ffa00008f70 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c1d82f0 con 0x7ffa000120a0 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.768+0000 7ffa23115640 10 monclient(hunting): authenticate will time out at 6965.066707s 2024-03-25T18:05:20.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.782+0000 7ffa1bfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ffa00008f70 0x7ffa1c0abbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:57280/0 (socket says 172.21.15.84:57280) 2024-03-25T18:05:20.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.782+0000 7ffa1bfff640 1 -- 172.21.15.84:0/452389160 learned_addr learned my addr 172.21.15.84:0/452389160 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:20.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.783+0000 7ffa197fa640 10 client.?.objecter ms_handle_connect 0x7ffa00008f70 2024-03-25T18:05:20.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.783+0000 7ffa197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:20.789 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.789+0000 7ffa197fa640 10 client.?.objecter ms_handle_connect 0x7ffa000056b0 2024-03-25T18:05:20.789 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.789+0000 7ffa197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:20.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.796+0000 7ffa197fa640 10 client.?.objecter ms_handle_connect 0x7ffa000120a0 2024-03-25T18:05:20.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.796+0000 7ffa197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047586662 0 0) 0x7ffa1c1d82f0 con 0x7ffa000120a0 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:05:20.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.842+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00015630 con 0x7ffa000120a0 2024-03-25T18:05:20.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1959032932 0 0) 0x7ffa00015630 con 0x7ffa000120a0 2024-03-25T18:05:20.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa1c1d82f0 con 0x7ffa000120a0 2024-03-25T18:05:20.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffa140048f0 con 0x7ffa000120a0 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 10 monclient(hunting): dump: 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:20.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.874+0000 7ffa197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:20.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384971973 0 0) 0x7ffa1c1d82f0 con 0x7ffa000120a0 2024-03-25T18:05:20.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 >> v1:172.21.15.157:6789/0 conn(0x7ffa00008f70 legacy=0x7ffa1c0abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:20.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 >> v1:172.21.15.84:6789/0 conn(0x7ffa000056b0 legacy=0x7ffa1c1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:20.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 10 monclient: _finish_hunting 0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 1 monclient: found mon.c 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c1d8480 con 0x7ffa000120a0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 10 monclient: _finish_auth 0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 10 monclient: _check_auth_tickets 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 5 monclient: authenticate success, global_id 4115 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 10 monclient: _renew_subs 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c140550 con 0x7ffa000120a0 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 10 monclient: _renew_subs 2024-03-25T18:05:20.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c0ac4b0 con 0x7ffa000120a0 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.935+0000 7ffa23115640 1 librados: init done 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.936+0000 7ffa23115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.936+0000 7ffa23115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.936+0000 7ffa23115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:20.936+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c0ac7b0 con 0x7ffa000120a0 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa14004d00 con 0x7ffa000120a0 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffa14004fb0 con 0x7ffa000120a0 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: dump: 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:21.112 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffa14005560 con 0x7ffa000120a0 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa1affd640 4 set_mon_vals no callback set 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 595425671 0 0) 0x7ffa14006b70 con 0x7ffa000120a0 2024-03-25T18:05:21.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 client.4115.objecter ms_dispatch 0x7ffa1c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 3 client.4115.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 3 client.4115.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 20 client.4115.objecter dump_active .. 0 homeless 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffa14018dc0 con 0x7ffa000120a0 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:21.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.112+0000 7ffa197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:21.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.181+0000 7ffa23115640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:05:21.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.181+0000 7ffa23115640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:05:21.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.181+0000 7ffa23115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:21.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.181+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ffa1c04f730 con 0x7ffa000120a0 2024-03-25T18:05:21.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.367+0000 7ffa197fa640 1 -- 172.21.15.84:0/452389160 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7ffa14005ad0 con 0x7ffa000120a0 2024-03-25T18:05:21.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.367+0000 7ffa197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:05:21.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.367+0000 7ffa197fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:05:21.368 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-03-25T18:05:21.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 10 librados: watch_flush enter 2024-03-25T18:05:21.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 10 librados: watch_flush exit 2024-03-25T18:05:21.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 20 client.4115.objecter shutdown clearing up homeless session... 2024-03-25T18:05:21.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 10 client.4115.objecter successfully canceled tick 2024-03-25T18:05:21.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 10 monclient: shutdown 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 >> v1:172.21.15.84:6790/0 conn(0x7ffa000120a0 legacy=0x7ffa1c1e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 shutdown_connections 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.368+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 >> 172.21.15.84:0/452389160 conn(0x7ffa1c0b7f40 msgr2=0x7ffa1c0b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.369+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 shutdown_connections 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.369+0000 7ffa23115640 1 -- 172.21.15.84:0/452389160 wait complete. 2024-03-25T18:05:21.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:21.369+0000 7ffa23115640 1 librados: shutdown 2024-03-25T18:05:21.380 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:05:21.380 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:05:21.381 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:21.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:05:21.396 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:21.395+0000 7fdc2ee3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:05:21.407 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:21.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:05:21.411 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:21.412+0000 7ff0bae0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:05:21.438 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:21.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:05:21.450 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:21.450+0000 7fdc2ee3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:05:21.475 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:21.475+0000 7ff0bae0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:05:21.502 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-25T18:05:21.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:05:21.570 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:21.570 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:05:21.598 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:21.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:05:21.665 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:21.665 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:05:21.733 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-25T18:05:21.733 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:05:21.800 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 784c9617-9da6-43f0-943e-f01177b02bba 0 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.941+0000 7f828b740640 10 monclient: get_monmap_and_config 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.941+0000 7f828b740640 10 monclient: build_initial_monmap 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.941+0000 7f828b740640 10 monclient: monmap: 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:21.942665+0000 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:21.942665+0000 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:21.942 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:21.943 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:21.943 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.942+0000 7f828b740640 1 Processor -- start 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.942+0000 7f828b740640 1 -- start start 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.942+0000 7f828b740640 10 monclient: init 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.942+0000 7f828b740640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient(hunting): picked mon.noname-a con 0x7f82840b4700 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient(hunting): picked mon.noname-b con 0x7f8284142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient(hunting): picked mon.noname-c con 0x7f82840a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:21.944 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840a9d60 con 0x7f82840b4700 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828413ff10 con 0x7f82840a96b0 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840aa1f0 con 0x7f8284142a20 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f828b740640 10 monclient(hunting): authenticate will time out at 6922.208757s 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f8289cdd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f82840a96b0 0x7f828413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:37326/0 (socket says 172.21.15.157:37326) 2024-03-25T18:05:21.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.943+0000 7f8289cdd640 1 -- 172.21.15.157:0/91764758 learned_addr learned my addr 172.21.15.157:0/91764758 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551133142 0 0) 0x7f828413ff10 con 0x7f82840a96b0 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:05:21.946 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.945+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8270003520 con 0x7f82840a96b0 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243123799 0 0) 0x7f82840aa1f0 con 0x7f8284142a20 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.946+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828413ff10 con 0x7f8284142a20 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364333522 0 0) 0x7f82840a9d60 con 0x7f82840b4700 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:21.947 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:21.948 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:21.948 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:05:21.948 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.947+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82840aa1f0 con 0x7f82840b4700 2024-03-25T18:05:21.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 437277706 0 0) 0x7f8270003520 con 0x7f82840a96b0 2024-03-25T18:05:21.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 >> v1:172.21.15.84:6790/0 conn(0x7f8284142a20 legacy=0x7f828413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 >> v1:172.21.15.84:6789/0 conn(0x7f82840b4700 legacy=0x7f8284142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:21.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 monclient: found mon.noname-c 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840b4ae0 con 0x7f82840a96b0 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: _finish_auth 0 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f827c001a10 con 0x7f82840a96b0 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f827bfff640 10 monclient: dump: 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.951 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f828b740640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:05:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.949+0000 7f828b740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f827c001b70 con 0x7f82840a96b0 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 1 -- 172.21.15.157:0/91764758 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f827c001f80 con 0x7f82840a96b0 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f827bfff640 10 monclient: dump: 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:21.977 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f828b740640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:21.978 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.976+0000 7f828b740640 4 set_mon_vals no callback set 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 10 monclient: shutdown 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 1 -- 172.21.15.157:0/91764758 >> v1:172.21.15.157:6789/0 conn(0x7f82840a96b0 legacy=0x7f828413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 1 -- 172.21.15.157:0/91764758 shutdown_connections 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 1 -- 172.21.15.157:0/91764758 >> 172.21.15.157:0/91764758 conn(0x7f82840668e0 msgr2=0x7f8284066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 1 -- 172.21.15.157:0/91764758 shutdown_connections 2024-03-25T18:05:21.980 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.977+0000 7f828b740640 1 -- 172.21.15.157:0/91764758 wait complete. 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 10 monclient: build_initial_monmap 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 10 monclient: monmap: 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 librados: starting msgr at 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 librados: starting objecter 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 Processor -- start 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 -- start start 2024-03-25T18:05:21.981 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 librados: setting wanted keys 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 1 librados: calling monclient init 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.978+0000 7f828b740640 10 monclient: init 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient(hunting): picked mon.noname-a con 0x7f82840a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient(hunting): picked mon.noname-b con 0x7f82840b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient(hunting): picked mon.noname-c con 0x7f8284142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840af020 con 0x7f82840a96b0 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840ae210 con 0x7f82840b4700 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840ae4f0 con 0x7f8284142a20 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:21.982 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.979+0000 7f828b740640 10 monclient(hunting): authenticate will time out at 6922.245054s 2024-03-25T18:05:21.983 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.981+0000 7f8288cdb640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f82840b4700 legacy=0x7f82840acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:05:21.983 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.981+0000 7f8288cdb640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f82840b4700 legacy=0x7f82840acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:05:21.983 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.981+0000 7f8288cdb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f82840b4700 0x7f82840acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T18:05:21.983 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.981+0000 7f8288cdb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f82840b4700 0x7f82840acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:05:21.983 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:21.981+0000 7f8279ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.157:6789/0 2024-03-25T18:05:22.007 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.006+0000 7fdc2ee3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:05:22.017 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.017+0000 7ff0bae0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:05:22.046 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.046+0000 7f8289cdd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8284142a20 0x7f82841e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.157:33742/0 (socket says 172.21.15.157:33742) 2024-03-25T18:05:22.046 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.046+0000 7f8289cdd640 1 -- 172.21.15.157:0/3820661034 learned_addr learned my addr 172.21.15.157:0/3820661034 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:22.046 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.046+0000 7f8279ffb640 10 client.?.objecter ms_handle_connect 0x7f8284142a20 2024-03-25T18:05:22.047 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.046+0000 7f8279ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953837088 0 0) 0x7f82840ae4f0 con 0x7f8284142a20 2024-03-25T18:05:22.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:05:22.050 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268004eb0 con 0x7f8284142a20 2024-03-25T18:05:22.050 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 client.?.objecter ms_handle_connect 0x7f82840a96b0 2024-03-25T18:05:22.050 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.049+0000 7f8279ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.052 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384847218 0 0) 0x7f82840af020 con 0x7f82840a96b0 2024-03-25T18:05:22.052 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.052 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.052 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.052 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:05:22.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.052+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82840ae4f0 con 0x7f82840a96b0 2024-03-25T18:05:22.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.053+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2896829475 0 0) 0x7f8268004eb0 con 0x7f8284142a20 2024-03-25T18:05:22.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82840af020 con 0x7f8284142a20 2024-03-25T18:05:22.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f827c001ec0 con 0x7f8284142a20 2024-03-25T18:05:22.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 10 monclient(hunting): dump: 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.055 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.054+0000 7f8279ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1490417966 0 0) 0x7f82840af020 con 0x7f8284142a20 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 >> v1:172.21.15.157:6789/0 conn(0x7f82840b4700 legacy=0x7f82840acc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 >> v1:172.21.15.84:6789/0 conn(0x7f82840a96b0 legacy=0x7f82840ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 1 monclient: found mon.c 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840ac0e0 con 0x7f8284142a20 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:22.058 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f8279ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 10 monclient: _renew_subs 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840ad340 con 0x7f8284142a20 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 10 monclient: _renew_subs 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f828413b070 con 0x7f8284142a20 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.057+0000 7f828b740640 1 librados: init done 2024-03-25T18:05:22.059 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.058+0000 7f828b740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.060 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.058+0000 7f828b740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.060 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.058+0000 7f828b740640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.060 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.058+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828404f730 con 0x7f8284142a20 2024-03-25T18:05:22.061 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.061+0000 7fdc2ee3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:05:22.075 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.075+0000 7ff0bae0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:05:22.106 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.105+0000 7fdc2ee3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:05:22.121 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.121+0000 7ff0bae0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f827c004e10 con 0x7f8284142a20 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f827c005120 con 0x7f8284142a20 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 10 monclient: dump: 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.157 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.156+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f827c005610 con 0x7f8284142a20 2024-03-25T18:05:22.158 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.157+0000 7f827b7fe640 4 set_mon_vals no callback set 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1121533457 0 0) 0x7f827c006910 con 0x7f8284142a20 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 10 client.4121.objecter ms_dispatch 0x7f8284000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 3 client.4121.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 3 client.4121.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f827c005980 con 0x7f8284142a20 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.158+0000 7f8279ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:22.160 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.159+0000 7fdc2ee3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:05:22.177 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.177+0000 7ff0bae0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:05:22.203 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.203+0000 7fdc2ee3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:05:22.219 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.219+0000 7ff0bae0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:05:22.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.229+0000 7f828b740640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "784c9617-9da6-43f0-943e-f01177b02bba", "id": 0}] 2024-03-25T18:05:22.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.229+0000 7f828b740640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "784c9617-9da6-43f0-943e-f01177b02bba", "id": 0}] 2024-03-25T18:05:22.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.229+0000 7f828b740640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.229+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd new", "uuid": "784c9617-9da6-43f0-943e-f01177b02bba", "id": 0} v 0) v1 -- 0x7f82840abec0 con 0x7f8284142a20 2024-03-25T18:05:22.319 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.319+0000 7f8279ffb640 1 -- 172.21.15.157:0/3820661034 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "784c9617-9da6-43f0-943e-f01177b02bba", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1711162288 0 732707515) 0x7f827c005c90 con 0x7f8284142a20 2024-03-25T18:05:22.319 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.319+0000 7f8279ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "784c9617-9da6-43f0-943e-f01177b02bba", "id": 0}] 2024-03-25T18:05:22.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.319+0000 7f8279ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:22.320 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 10 librados: watch_flush enter 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 10 librados: watch_flush exit 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 10 monclient: shutdown 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 >> v1:172.21.15.84:6790/0 conn(0x7f8284142a20 legacy=0x7f82841e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 shutdown_connections 2024-03-25T18:05:22.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 >> 172.21.15.157:0/3820661034 conn(0x7f82840668e0 msgr2=0x7f82840b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:22.321 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.320+0000 7fdc2ee3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:05:22.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 shutdown_connections 2024-03-25T18:05:22.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 -- 172.21.15.157:0/3820661034 wait complete. 2024-03-25T18:05:22.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.320+0000 7f828b740640 1 librados: shutdown 2024-03-25T18:05:22.331 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9d216077-d072-4e6a-a53f-5f0c81125515 1 2024-03-25T18:05:22.339 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.340+0000 7ff0bae0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:05:22.414 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.413+0000 7fdc2ee3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:05:22.414 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.414+0000 7ff0bae0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.431+0000 7f6eb9231640 10 monclient: get_monmap_and_config 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.431+0000 7f6eb9231640 10 monclient: build_initial_monmap 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.431+0000 7f6eb9231640 10 monclient: monmap: 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:22.432546+0000 2024-03-25T18:05:22.432 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:22.432546+0000 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 1 Processor -- start 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 1 -- start start 2024-03-25T18:05:22.433 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient: init 2024-03-25T18:05:22.434 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:22.434 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:22.434 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-b con 0x7f6eb40b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:22.434 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-c con 0x7f6eb4142a20 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:22.434 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-a con 0x7f6eb40a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40a9d60 con 0x7f6eb40a96b0 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb413ff10 con 0x7f6eb4142a20 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40aa1f0 con 0x7f6eb40b4700 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.432+0000 7f6eb9231640 10 monclient(hunting): authenticate will time out at 6922.698557s 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.433+0000 7f6eb259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6eb4142a20 0x7f6eb413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:37348/0 (socket says 172.21.15.157:37348) 2024-03-25T18:05:22.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.433+0000 7f6eb259c640 1 -- 172.21.15.157:0/3669743681 learned_addr learned my addr 172.21.15.157:0/3669743681 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:22.436 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162878625 0 0) 0x7f6eb40aa1f0 con 0x7f6eb40b4700 2024-03-25T18:05:22.436 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9c003520 con 0x7f6eb40b4700 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922380227 0 0) 0x7f6eb413ff10 con 0x7f6eb4142a20 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.435+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb40aa1f0 con 0x7f6eb4142a20 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355038835 0 0) 0x7f6eb40a9d60 con 0x7f6eb40a96b0 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.437 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:05:22.438 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.436+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb413ff10 con 0x7f6eb40a96b0 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406855544 0 0) 0x7f6e9c003520 con 0x7f6eb40b4700 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 >> v1:172.21.15.157:6789/0 conn(0x7f6eb4142a20 legacy=0x7f6eb413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 >> v1:172.21.15.84:6789/0 conn(0x7f6eb40a96b0 legacy=0x7f6eb413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40b4ae0 con 0x7f6eb40b4700 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:22.440 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ea4001690 con 0x7f6eb40b4700 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb1d9b640 10 monclient: dump: 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb9231640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:05:22.441 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.439+0000 7f6eb9231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea40017f0 con 0x7f6eb40b4700 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 1 -- 172.21.15.157:0/3669743681 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ea4001c00 con 0x7f6eb40b4700 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb1d9b640 10 monclient: dump: 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.443 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb9231640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.440+0000 7f6eb9231640 4 set_mon_vals no callback set 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 10 monclient: shutdown 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 1 -- 172.21.15.157:0/3669743681 >> v1:172.21.15.84:6790/0 conn(0x7f6eb40b4700 legacy=0x7f6eb4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 1 -- 172.21.15.157:0/3669743681 shutdown_connections 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 1 -- 172.21.15.157:0/3669743681 >> 172.21.15.157:0/3669743681 conn(0x7f6eb40668e0 msgr2=0x7f6eb4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 1 -- 172.21.15.157:0/3669743681 shutdown_connections 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.441+0000 7f6eb9231640 1 -- 172.21.15.157:0/3669743681 wait complete. 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.442+0000 7f6eb9231640 10 monclient: build_initial_monmap 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.442+0000 7f6eb9231640 10 monclient: monmap: 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:22.444 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.442+0000 7f6eb9231640 1 librados: starting msgr at 2024-03-25T18:05:22.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.442+0000 7f6eb9231640 1 librados: starting objecter 2024-03-25T18:05:22.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 Processor -- start 2024-03-25T18:05:22.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 -- start start 2024-03-25T18:05:22.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 librados: setting wanted keys 2024-03-25T18:05:22.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 librados: calling monclient init 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient: init 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-a con 0x7f6eb40a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-b con 0x7f6eb40b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient(hunting): picked mon.noname-c con 0x7f6eb4142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb41db950 con 0x7f6eb40a96b0 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb41daa40 con 0x7f6eb40b4700 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb41dad20 con 0x7f6eb4142a20 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.443+0000 7f6eb9231640 10 monclient(hunting): authenticate will time out at 6922.709322s 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6eb259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6eb40b4700 0x7f6eb41d47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:37358/0 (socket says 172.21.15.157:37358) 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6eb259c640 1 -- 172.21.15.157:0/3034091591 learned_addr learned my addr 172.21.15.157:0/3034091591 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6eb40b4700 2024-03-25T18:05:22.448 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6eb4142a20 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6eb40a96b0 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.444+0000 7f6e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464960794 0 0) 0x7f6eb41dad20 con 0x7f6eb4142a20 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea0004eb0 con 0x7f6eb4142a20 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004149526 0 0) 0x7f6eb41daa40 con 0x7f6eb40b4700 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb41dad20 con 0x7f6eb40b4700 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740466043 0 0) 0x7f6eb41db950 con 0x7f6eb40a96b0 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:05:22.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.446+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb41daa40 con 0x7f6eb40a96b0 2024-03-25T18:05:22.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.450+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428885913 0 0) 0x7f6ea0004eb0 con 0x7f6eb4142a20 2024-03-25T18:05:22.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.450+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb41db950 con 0x7f6eb4142a20 2024-03-25T18:05:22.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748728564 0 0) 0x7f6eb41dad20 con 0x7f6eb40b4700 2024-03-25T18:05:22.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea0004eb0 con 0x7f6eb40b4700 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ea8001970 con 0x7f6eb40b4700 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6eac001c10 con 0x7f6eb4142a20 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.451+0000 7f6e9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.452+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809457753 0 0) 0x7f6eb41daa40 con 0x7f6eb40a96b0 2024-03-25T18:05:22.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.452+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb41dad20 con 0x7f6eb40a96b0 2024-03-25T18:05:22.456 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1063939991 0 0) 0x7f6eb41db950 con 0x7f6eb4142a20 2024-03-25T18:05:22.456 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 >> v1:172.21.15.157:6789/0 conn(0x7f6eb40b4700 legacy=0x7f6eb41d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.456 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 >> v1:172.21.15.84:6789/0 conn(0x7f6eb40a96b0 legacy=0x7f6eb41db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 1 monclient: found mon.c 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40abc50 con 0x7f6eb4142a20 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 10 monclient: _renew_subs 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb41d4f00 con 0x7f6eb4142a20 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 10 monclient: _renew_subs 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb40ad000 con 0x7f6eb4142a20 2024-03-25T18:05:22.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.455+0000 7f6eb9231640 1 librados: init done 2024-03-25T18:05:22.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.456+0000 7f6eb9231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.456+0000 7f6eb9231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.456+0000 7f6eb9231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.456+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb404f730 con 0x7f6eb4142a20 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eac003990 con 0x7f6eb4142a20 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6eac003ca0 con 0x7f6eb4142a20 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 10 monclient: dump: 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.462 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6eac004190 con 0x7f6eb4142a20 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6eb159a640 4 set_mon_vals no callback set 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3010261942 0 0) 0x7f6eac005730 con 0x7f6eb4142a20 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 10 client.4127.objecter ms_dispatch 0x7f6eb4000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 3 client.4127.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 3 client.4127.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:05:22.463 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.461+0000 7f6e9b7fe640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T18:05:22.463 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.463+0000 7fdc2ee3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:05:22.465 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.465+0000 7ff0bae0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:05:22.516 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.515+0000 7fdc2ee3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:05:22.519 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.519+0000 7ff0bae0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:05:22.556 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.556+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6eac005a50 con 0x7f6eb4142a20 2024-03-25T18:05:22.556 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.556+0000 7f6e9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.556 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.556+0000 7f6e9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:22.562 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.562+0000 7fdc2ee3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:05:22.567 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.568+0000 7ff0bae0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:05:22.626 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.626+0000 7f6eb9231640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9d216077-d072-4e6a-a53f-5f0c81125515", "id": 1}] 2024-03-25T18:05:22.627 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.626+0000 7f6eb9231640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9d216077-d072-4e6a-a53f-5f0c81125515", "id": 1}] 2024-03-25T18:05:22.627 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.626+0000 7f6eb9231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:22.627 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.626+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9d216077-d072-4e6a-a53f-5f0c81125515", "id": 1} v 0) v1 -- 0x7f6eb4142220 con 0x7f6eb4142a20 2024-03-25T18:05:22.658 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.657+0000 7f6e9b7fe640 1 -- 172.21.15.157:0/3034091591 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d216077-d072-4e6a-a53f-5f0c81125515", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1437686525 0 940483276) 0x7f6eac004440 con 0x7f6eb4142a20 2024-03-25T18:05:22.658 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.657+0000 7f6e9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9d216077-d072-4e6a-a53f-5f0c81125515", "id": 1}] 2024-03-25T18:05:22.658 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.657+0000 7f6e9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:22.658 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-03-25T18:05:22.659 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 10 librados: watch_flush enter 2024-03-25T18:05:22.659 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 10 librados: watch_flush exit 2024-03-25T18:05:22.659 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 10 client.4127.objecter successfully canceled tick 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 10 monclient: shutdown 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 >> v1:172.21.15.84:6790/0 conn(0x7f6eb4142a20 legacy=0x7f6eb40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 shutdown_connections 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 >> 172.21.15.157:0/3034091591 conn(0x7f6eb40668e0 msgr2=0x7f6eb4066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 shutdown_connections 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 -- 172.21.15.157:0/3034091591 wait complete. 2024-03-25T18:05:22.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.658+0000 7f6eb9231640 1 librados: shutdown 2024-03-25T18:05:22.669 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new ac1039e9-08a7-47d6-9e2a-571cfab09efd 2 2024-03-25T18:05:22.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 10 monclient: get_monmap_and_config 2024-03-25T18:05:22.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 10 monclient: build_initial_monmap 2024-03-25T18:05:22.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 10 monclient: monmap: 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:22.771049+0000 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:22.771049+0000 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:22.771 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.772 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 1 Processor -- start 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 1 -- start start 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.770+0000 7fa22a154640 10 monclient: init 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-c con 0x7fa2240b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-b con 0x7fa224142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-a con 0x7fa2240a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240a9d60 con 0x7fa2240a96b0 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22413ff10 con 0x7fa2240b4700 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240aa1f0 con 0x7fa224142a20 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa22a154640 10 monclient(hunting): authenticate will time out at 6923.037138s 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa2237fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa2240b4700 0x7fa224142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42238/0 (socket says 172.21.15.157:42238) 2024-03-25T18:05:22.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.771+0000 7fa2237fe640 1 -- 172.21.15.157:0/3775757639 learned_addr learned my addr 172.21.15.157:0/3775757639 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569813190 0 0) 0x7fa22413ff10 con 0x7fa2240b4700 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c003520 con 0x7fa2240b4700 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780119321 0 0) 0x7fa2240a9d60 con 0x7fa2240a96b0 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.774+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa22413ff10 con 0x7fa2240a96b0 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488184994 0 0) 0x7fa2240aa1f0 con 0x7fa224142a20 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:05:22.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.775+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2240a9d60 con 0x7fa224142a20 2024-03-25T18:05:22.778 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679471899 0 0) 0x7fa20c003520 con 0x7fa2240b4700 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 >> v1:172.21.15.84:6790/0 conn(0x7fa224142a20 legacy=0x7fa22413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 >> v1:172.21.15.84:6789/0 conn(0x7fa2240a96b0 legacy=0x7fa22413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 monclient: found mon.noname-c 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b4ae0 con 0x7fa2240b4700 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa218001690 con 0x7fa2240b4700 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa22a154640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa22a154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:22.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa2227fc640 10 monclient: dump: 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.777+0000 7fa22a154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:22.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2180017f0 con 0x7fa2240b4700 2024-03-25T18:05:22.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 1 -- 172.21.15.157:0/3775757639 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa218001c00 con 0x7fa2240b4700 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 4 set_mon_vals no callback set 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa2227fc640 10 monclient: dump: 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.782 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 10 monclient: shutdown 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 1 -- 172.21.15.157:0/3775757639 >> v1:172.21.15.157:6789/0 conn(0x7fa2240b4700 legacy=0x7fa224142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.780+0000 7fa22a154640 1 -- 172.21.15.157:0/3775757639 shutdown_connections 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.781+0000 7fa22a154640 1 -- 172.21.15.157:0/3775757639 >> 172.21.15.157:0/3775757639 conn(0x7fa2240668e0 msgr2=0x7fa224066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.781+0000 7fa22a154640 1 -- 172.21.15.157:0/3775757639 shutdown_connections 2024-03-25T18:05:22.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.781+0000 7fa22a154640 1 -- 172.21.15.157:0/3775757639 wait complete. 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.781+0000 7fa22a154640 10 monclient: build_initial_monmap 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.781+0000 7fa22a154640 10 monclient: monmap: 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:22.785 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 librados: starting msgr at 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 librados: starting objecter 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 Processor -- start 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 -- start start 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 librados: setting wanted keys 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 1 librados: calling monclient init 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.782+0000 7fa22a154640 10 monclient: init 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-a con 0x7fa2240b4700 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient(hunting): picked mon.noname-c con 0x7fa224142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b1850 con 0x7fa2240b4700 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240ad150 con 0x7fa2240a96b0 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240ad430 con 0x7fa224142a20 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.783+0000 7fa22a154640 10 monclient(hunting): authenticate will time out at 6923.049181s 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa2237fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa2240a96b0 0x7fa224139db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42252/0 (socket says 172.21.15.157:42252) 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa222ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa2240b4700 0x7fa2240ad850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.157:42018/0 (socket says 172.21.15.157:42018) 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa2237fe640 1 -- 172.21.15.157:0/75794172 learned_addr learned my addr 172.21.15.157:0/75794172 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa2240b4700 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa2240a96b0 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.787 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa224142a20 2024-03-25T18:05:22.788 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.784+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:22.788 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521956859 0 0) 0x7fa2240b1850 con 0x7fa2240b4700 2024-03-25T18:05:22.788 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.788 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.788 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa210004eb0 con 0x7fa2240b4700 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263932998 0 0) 0x7fa2240ad150 con 0x7fa2240a96b0 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.786+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2240b1850 con 0x7fa2240a96b0 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158530586 0 0) 0x7fa2240ad430 con 0x7fa224142a20 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:22.789 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:22.790 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:22.790 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:05:22.790 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.789+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2240ad150 con 0x7fa224142a20 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383275027 0 0) 0x7fa2240b1850 con 0x7fa2240a96b0 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2240ad430 con 0x7fa2240a96b0 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa218001b40 con 0x7fa2240a96b0 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): dump: 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.792 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4098472002 0 0) 0x7fa210004eb0 con 0x7fa2240b4700 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2240b1850 con 0x7fa2240b4700 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa214001970 con 0x7fa2240b4700 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.791+0000 7fa20bfff640 10 monclient(hunting): dump: 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.794 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4114305687 0 0) 0x7fa2240ad430 con 0x7fa2240a96b0 2024-03-25T18:05:22.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 >> v1:172.21.15.84:6790/0 conn(0x7fa224142a20 legacy=0x7fa2241e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 >> v1:172.21.15.84:6789/0 conn(0x7fa2240b4700 legacy=0x7fa2240ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:22.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 1 monclient: found mon.b 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240ac180 con 0x7fa2240a96b0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 10 monclient: _finish_auth 0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa20bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 10 monclient: _renew_subs 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22413b5c0 con 0x7fa2240a96b0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 10 monclient: _renew_subs 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2240abaf0 con 0x7fa2240a96b0 2024-03-25T18:05:22.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.794+0000 7fa22a154640 1 librados: init done 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.795+0000 7fa22a154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.795+0000 7fa22a154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.795+0000 7fa22a154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.795+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22404f730 con 0x7fa2240a96b0 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.796+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2180011f0 con 0x7fa2240a96b0 2024-03-25T18:05:22.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.796+0000 7fa20bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:22.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.796+0000 7fa221ffb640 4 set_mon_vals no callback set 2024-03-25T18:05:22.956 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:22.955+0000 7fdc2ee3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:05:22.966 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:22.967+0000 7ff0bae0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa218004c30 con 0x7fa2240a96b0 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 monclient: dump: 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:22.991 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa2180051a0 con 0x7fa2240a96b0 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3019041731 0 0) 0x7fa2180067c0 con 0x7fa2240a96b0 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 client.4129.objecter ms_dispatch 0x7fa224000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa218006b60 con 0x7fa2240a96b0 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:22.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:22.990+0000 7fa20bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:23.063 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.062+0000 7fa22a154640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac1039e9-08a7-47d6-9e2a-571cfab09efd", "id": 2}] 2024-03-25T18:05:23.063 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.063+0000 7fa22a154640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac1039e9-08a7-47d6-9e2a-571cfab09efd", "id": 2}] 2024-03-25T18:05:23.063 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.063+0000 7fa22a154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:23.063 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.063+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ac1039e9-08a7-47d6-9e2a-571cfab09efd", "id": 2} v 0) v1 -- 0x7fa2240b7190 con 0x7fa2240a96b0 2024-03-25T18:05:23.103 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.103+0000 7ff0bae0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:05:23.112 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.112+0000 7fdc2ee3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:05:23.190 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.190+0000 7fa20bfff640 1 -- 172.21.15.157:0/75794172 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac1039e9-08a7-47d6-9e2a-571cfab09efd", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3465766559 0 216599125) 0x7fa218006470 con 0x7fa2240a96b0 2024-03-25T18:05:23.191 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.190+0000 7fa20bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac1039e9-08a7-47d6-9e2a-571cfab09efd", "id": 2}] 2024-03-25T18:05:23.191 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.190+0000 7fa20bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:23.191 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 10 librados: watch_flush enter 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 10 librados: watch_flush exit 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 10 monclient: shutdown 2024-03-25T18:05:23.192 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 >> v1:172.21.15.157:6789/0 conn(0x7fa2240a96b0 legacy=0x7fa224139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 shutdown_connections 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 >> 172.21.15.157:0/75794172 conn(0x7fa2240668e0 msgr2=0x7fa2241364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 shutdown_connections 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 -- 172.21.15.157:0/75794172 wait complete. 2024-03-25T18:05:23.193 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.191+0000 7fa22a154640 1 librados: shutdown 2024-03-25T18:05:23.202 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new c1ca3152-d5fb-4419-918f-0ed67b43351c 3 2024-03-25T18:05:23.235 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.234+0000 7fdc2ee3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:05:23.283 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.282+0000 7fdc2ee3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.300+0000 7f9e59d8d640 10 monclient: get_monmap_and_config 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.300+0000 7f9e59d8d640 10 monclient: build_initial_monmap 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.300+0000 7f9e59d8d640 10 monclient: monmap: 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:23.301744+0000 2024-03-25T18:05:23.301 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:23.301744+0000 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:23.302 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.301+0000 7f9e59d8d640 1 Processor -- start 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.301+0000 7f9e59d8d640 1 -- start start 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.301+0000 7f9e59d8d640 10 monclient: init 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e540b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e54142a20 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e540a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e540a9d60 con 0x7f9e54142a20 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5413ff10 con 0x7f9e540a96b0 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e540aa1f0 con 0x7f9e540b4700 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e59d8d640 10 monclient(hunting): authenticate will time out at 6923.567976s 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e53fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9e540a96b0 0x7f9e5413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42264/0 (socket says 172.21.15.157:42264) 2024-03-25T18:05:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.302+0000 7f9e53fff640 1 -- 172.21.15.157:0/180649932 learned_addr learned my addr 172.21.15.157:0/180649932 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238108043 0 0) 0x7f9e5413ff10 con 0x7f9e540a96b0 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e3c003520 con 0x7f9e540a96b0 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250353448 0 0) 0x7f9e540aa1f0 con 0x7f9e540b4700 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5413ff10 con 0x7f9e540b4700 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467012661 0 0) 0x7f9e540a9d60 con 0x7f9e54142a20 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:05:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.305+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e540aa1f0 con 0x7f9e54142a20 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1701099385 0 0) 0x7f9e3c003520 con 0x7f9e540a96b0 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 >> v1:172.21.15.84:6790/0 conn(0x7f9e540b4700 legacy=0x7f9e54142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 >> v1:172.21.15.84:6789/0 conn(0x7f9e54142a20 legacy=0x7f9e5413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 monclient: found mon.noname-c 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540b4ae0 con 0x7f9e540a96b0 2024-03-25T18:05:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e4c001690 con 0x7f9e540a96b0 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e59d8d640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e527fc640 10 monclient: dump: 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.308+0000 7f9e59d8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e4c0017f0 con 0x7f9e540a96b0 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 1 -- 172.21.15.157:0/180649932 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e4c001c00 con 0x7f9e540a96b0 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e59d8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e59d8d640 4 set_mon_vals no callback set 2024-03-25T18:05:23.314 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.310+0000 7f9e527fc640 10 monclient: dump: 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.315 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 10 monclient: shutdown 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 1 -- 172.21.15.157:0/180649932 >> v1:172.21.15.157:6789/0 conn(0x7f9e540a96b0 legacy=0x7f9e5413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 1 -- 172.21.15.157:0/180649932 shutdown_connections 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 1 -- 172.21.15.157:0/180649932 >> 172.21.15.157:0/180649932 conn(0x7f9e540668e0 msgr2=0x7f9e54066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 1 -- 172.21.15.157:0/180649932 shutdown_connections 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 1 -- 172.21.15.157:0/180649932 wait complete. 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 10 monclient: build_initial_monmap 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.311+0000 7f9e59d8d640 10 monclient: monmap: 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:23.316 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 librados: starting msgr at 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 librados: starting objecter 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 Processor -- start 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 -- start start 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 librados: setting wanted keys 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 1 librados: calling monclient init 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 10 monclient: init 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.312+0000 7f9e59d8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:23.317 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e540a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e540b4700 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e54142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5413c830 con 0x7f9e540b4700 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e54133a90 con 0x7f9e540a96b0 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e540ac8f0 con 0x7f9e54142a20 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e59d8d640 10 monclient(hunting): authenticate will time out at 6923.578757s 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e537fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9e540a96b0 0x7f9e54138f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42278/0 (socket says 172.21.15.157:42278) 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e537fe640 1 -- 172.21.15.157:0/2733581220 learned_addr learned my addr 172.21.15.157:0/2733581220 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e3bfff640 10 client.?.objecter ms_handle_connect 0x7f9e540a96b0 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.313+0000 7f9e3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.314+0000 7f9e3bfff640 10 client.?.objecter ms_handle_connect 0x7f9e54142a20 2024-03-25T18:05:23.318 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.314+0000 7f9e3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410731200 0 0) 0x7f9e54133a90 con 0x7f9e540a96b0 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.315+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e40004eb0 con 0x7f9e540a96b0 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650202978 0 0) 0x7f9e540ac8f0 con 0x7f9e54142a20 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54133a90 con 0x7f9e54142a20 2024-03-25T18:05:23.320 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 client.?.objecter ms_handle_connect 0x7f9e540b4700 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.316+0000 7f9e3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088949522 0 0) 0x7f9e5413c830 con 0x7f9e540b4700 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:05:23.321 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.317+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e540ac8f0 con 0x7f9e540b4700 2024-03-25T18:05:23.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611696544 0 0) 0x7f9e40004eb0 con 0x7f9e540a96b0 2024-03-25T18:05:23.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e5413c830 con 0x7f9e540a96b0 2024-03-25T18:05:23.322 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e44001a30 con 0x7f9e540a96b0 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): dump: 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.323 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1828704689 0 0) 0x7f9e54133a90 con 0x7f9e54142a20 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e40004eb0 con 0x7f9e54142a20 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e4c001b00 con 0x7f9e54142a20 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.320+0000 7f9e3bfff640 10 monclient(hunting): dump: 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.324 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.325 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.325 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.325 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.326 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196684319 0 0) 0x7f9e540ac8f0 con 0x7f9e540b4700 2024-03-25T18:05:23.326 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e54133a90 con 0x7f9e540b4700 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e48001c30 con 0x7f9e540b4700 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.322+0000 7f9e3bfff640 10 monclient(hunting): dump: 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.327 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 701110055 0 0) 0x7f9e40004eb0 con 0x7f9e54142a20 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 >> v1:172.21.15.157:6789/0 conn(0x7f9e540a96b0 legacy=0x7f9e54138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 >> v1:172.21.15.84:6789/0 conn(0x7f9e540b4700 legacy=0x7f9e54132c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 1 monclient: found mon.c 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540ac610 con 0x7f9e54142a20 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 10 monclient: _finish_auth 0 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 5 monclient: authenticate success, global_id 4139 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 10 monclient: _renew_subs 2024-03-25T18:05:23.328 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5413ccc0 con 0x7f9e54142a20 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 10 monclient: _renew_subs 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e541397f0 con 0x7f9e54142a20 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.323+0000 7f9e59d8d640 1 librados: init done 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.324+0000 7f9e59d8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.324+0000 7f9e59d8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.324+0000 7f9e59d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.329 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.324+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5404f730 con 0x7f9e54142a20 2024-03-25T18:05:23.398 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.397+0000 7fdc2ee3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:05:23.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.415+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e4c0010a0 con 0x7f9e54142a20 2024-03-25T18:05:23.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.415+0000 7f9e3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:23.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.415+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9e4c004a90 con 0x7f9e54142a20 2024-03-25T18:05:23.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.415+0000 7f9e3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 10 monclient: dump: 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9e4c004fc0 con 0x7f9e54142a20 2024-03-25T18:05:23.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e51ffb640 4 set_mon_vals no callback set 2024-03-25T18:05:23.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 846357925 0 0) 0x7f9e4c0065b0 con 0x7f9e54142a20 2024-03-25T18:05:23.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 10 client.4139.objecter ms_dispatch 0x7f9e54000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:05:23.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 3 client.4139.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:05:23.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 3 client.4139.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:05:23.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T18:05:23.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9e4c0068e0 con 0x7f9e54142a20 2024-03-25T18:05:23.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.416+0000 7f9e3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:23.488 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.487+0000 7f9e59d8d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c1ca3152-d5fb-4419-918f-0ed67b43351c", "id": 3}] 2024-03-25T18:05:23.488 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.487+0000 7f9e59d8d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c1ca3152-d5fb-4419-918f-0ed67b43351c", "id": 3}] 2024-03-25T18:05:23.488 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.487+0000 7f9e59d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.488 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.487+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c1ca3152-d5fb-4419-918f-0ed67b43351c", "id": 3} v 0) v1 -- 0x7f9e540670b0 con 0x7f9e54142a20 2024-03-25T18:05:23.494 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.494+0000 7fdc2ee3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:05:23.496 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.496+0000 7ff0bae0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:05:23.533 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.533+0000 7f9e3bfff640 1 -- 172.21.15.157:0/2733581220 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c1ca3152-d5fb-4419-918f-0ed67b43351c", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3593288606 0 525046306) 0x7f9e4c0061f0 con 0x7f9e54142a20 2024-03-25T18:05:23.534 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.533+0000 7f9e3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c1ca3152-d5fb-4419-918f-0ed67b43351c", "id": 3}] 2024-03-25T18:05:23.534 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.533+0000 7f9e3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:23.534 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-03-25T18:05:23.535 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 10 librados: watch_flush enter 2024-03-25T18:05:23.535 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 10 librados: watch_flush exit 2024-03-25T18:05:23.535 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 10 client.4139.objecter successfully canceled tick 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 10 monclient: shutdown 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 >> v1:172.21.15.84:6790/0 conn(0x7f9e54142a20 legacy=0x7f9e54133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 shutdown_connections 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 >> 172.21.15.157:0/2733581220 conn(0x7f9e540668e0 msgr2=0x7f9e5413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 shutdown_connections 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 -- 172.21.15.157:0/2733581220 wait complete. 2024-03-25T18:05:23.536 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.534+0000 7f9e59d8d640 1 librados: shutdown 2024-03-25T18:05:23.546 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b 4 2024-03-25T18:05:23.548 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.548+0000 7ff0bae0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:05:23.658 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.657+0000 7fdc2ee3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:05:23.659 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.660+0000 7ff0bae0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:05:23.686 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.685+0000 7f815704a640 10 monclient: get_monmap_and_config 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.685+0000 7f815704a640 10 monclient: build_initial_monmap 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.685+0000 7f815704a640 10 monclient: monmap: 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:23.686802+0000 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:23.686802+0000 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:23.687 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:23.688 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:23.688 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.688 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.686+0000 7f815704a640 1 Processor -- start 2024-03-25T18:05:23.688 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.686+0000 7f815704a640 1 -- start start 2024-03-25T18:05:23.688 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.686+0000 7f815704a640 10 monclient: init 2024-03-25T18:05:23.689 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient(hunting): picked mon.noname-b con 0x7f81500b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient(hunting): picked mon.noname-a con 0x7f8150142a20 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient(hunting): picked mon.noname-c con 0x7f81500a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500a9d60 con 0x7f8150142a20 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815013ff10 con 0x7f81500a96b0 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500aa1f0 con 0x7f81500b4700 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f815704a640 10 monclient(hunting): authenticate will time out at 6923.953217s 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f81555e7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f81500a96b0 0x7f815013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42288/0 (socket says 172.21.15.157:42288) 2024-03-25T18:05:23.690 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.687+0000 7f81555e7640 1 -- 172.21.15.157:0/2377018162 learned_addr learned my addr 172.21.15.157:0/2377018162 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040789841 0 0) 0x7f815013ff10 con 0x7f81500a96b0 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8140003520 con 0x7f81500a96b0 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805505692 0 0) 0x7f81500a9d60 con 0x7f8150142a20 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f815013ff10 con 0x7f8150142a20 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458086967 0 0) 0x7f81500aa1f0 con 0x7f81500b4700 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:05:23.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.690+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81500a9d60 con 0x7f81500b4700 2024-03-25T18:05:23.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106321854 0 0) 0x7f815013ff10 con 0x7f8150142a20 2024-03-25T18:05:23.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 >> v1:172.21.15.84:6790/0 conn(0x7f81500b4700 legacy=0x7f8150142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 >> v1:172.21.15.157:6789/0 conn(0x7f81500a96b0 legacy=0x7f815013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:23.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81500b4ae0 con 0x7f8150142a20 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.692+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f813c001670 con 0x7f8150142a20 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.693+0000 7f814f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.693+0000 7f814f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.693+0000 7f814f7fe640 10 monclient: dump: 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.695 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.693+0000 7f815704a640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:05:23.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.693+0000 7f815704a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:23.703 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.702+0000 7fdc2ee3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:05:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f813c0017d0 con 0x7f8150142a20 2024-03-25T18:05:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 1 -- 172.21.15.157:0/2377018162 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f813c001be0 con 0x7f8150142a20 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f814f7fe640 10 monclient: dump: 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.724 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 4 set_mon_vals no callback set 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 10 monclient: shutdown 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 1 -- 172.21.15.157:0/2377018162 >> v1:172.21.15.84:6789/0 conn(0x7f8150142a20 legacy=0x7f815013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 1 -- 172.21.15.157:0/2377018162 shutdown_connections 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.722+0000 7f815704a640 1 -- 172.21.15.157:0/2377018162 >> 172.21.15.157:0/2377018162 conn(0x7f81500668e0 msgr2=0x7f8150066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 1 -- 172.21.15.157:0/2377018162 shutdown_connections 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 1 -- 172.21.15.157:0/2377018162 wait complete. 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 10 monclient: build_initial_monmap 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 10 monclient: monmap: 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 1 librados: starting msgr at 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.723+0000 7f815704a640 1 librados: starting objecter 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 Processor -- start 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 -- start start 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 librados: setting wanted keys 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 librados: calling monclient init 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient: init 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient(hunting): picked mon.noname-a con 0x7f81500a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient(hunting): picked mon.noname-c con 0x7f81500b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient(hunting): picked mon.noname-b con 0x7f8150142a20 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815013c060 con 0x7f81500a96b0 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815013ca90 con 0x7f8150142a20 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500aba90 con 0x7f81500b4700 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.724+0000 7f815704a640 10 monclient(hunting): authenticate will time out at 6923.990454s 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.725+0000 7f81555e7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f8150142a20 0x7f8150138d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42294/0 (socket says 172.21.15.157:42294) 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.725+0000 7f81555e7640 1 -- 172.21.15.157:0/1921224179 learned_addr learned my addr 172.21.15.157:0/1921224179 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.725+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f8150142a20 2024-03-25T18:05:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.725+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.726+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f81500b4700 2024-03-25T18:05:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.726+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f81500a96b0 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633393694 0 0) 0x7f81500aba90 con 0x7f81500b4700 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:05:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.728+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8134004eb0 con 0x7f81500b4700 2024-03-25T18:05:23.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505764793 0 0) 0x7f8134004eb0 con 0x7f81500b4700 2024-03-25T18:05:23.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81500aba90 con 0x7f81500b4700 2024-03-25T18:05:23.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f813c001ba0 con 0x7f81500b4700 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 10 monclient(hunting): dump: 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.746+0000 7f814d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:23.768 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.768+0000 7ff0bae0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125522768 0 0) 0x7f815013c060 con 0x7f81500a96b0 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:05:23.770 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.769+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8134004eb0 con 0x7f81500a96b0 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133205826 0 0) 0x7f815013ca90 con 0x7f8150142a20 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:05:23.773 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.772+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f815013c060 con 0x7f8150142a20 2024-03-25T18:05:23.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235221723 0 0) 0x7f8134004eb0 con 0x7f81500a96b0 2024-03-25T18:05:23.775 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f815013ca90 con 0x7f81500a96b0 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8144001830 con 0x7f81500a96b0 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.775+0000 7f814d7fa640 10 monclient(hunting): dump: 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.776 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.777 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.777 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.777 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.777+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674834706 0 0) 0x7f815013c060 con 0x7f8150142a20 2024-03-25T18:05:23.777 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.777+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8134004eb0 con 0x7f8150142a20 2024-03-25T18:05:23.779 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130996734 0 0) 0x7f81500aba90 con 0x7f81500b4700 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 >> v1:172.21.15.157:6789/0 conn(0x7f8150142a20 legacy=0x7f8150138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 >> v1:172.21.15.84:6789/0 conn(0x7f81500a96b0 legacy=0x7f815013b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 1 monclient: found mon.c 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815013af40 con 0x7f81500b4700 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f814d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:23.780 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f815704a640 5 monclient: authenticate success, global_id 4145 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f815704a640 10 monclient: _renew_subs 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f815704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.778+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815013b1f0 con 0x7f81500b4700 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 monclient: _renew_subs 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81501e2e40 con 0x7f81500b4700 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 1 librados: init done 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.779+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815004f730 con 0x7f81500b4700 2024-03-25T18:05:23.860 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:23.860+0000 7fdc2ee3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f813c0038a0 con 0x7f81500b4700 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f813c003bb0 con 0x7f81500b4700 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 10 monclient: dump: 2024-03-25T18:05:23.876 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f813c0040a0 con 0x7f81500b4700 2024-03-25T18:05:23.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.875+0000 7f814effd640 4 set_mon_vals no callback set 2024-03-25T18:05:23.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 145093442 0 0) 0x7f813c005670 con 0x7f81500b4700 2024-03-25T18:05:23.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 10 client.4145.objecter ms_dispatch 0x7f8150000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:05:23.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 3 client.4145.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:05:23.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 3 client.4145.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:05:23.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T18:05:23.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f813c005180 con 0x7f81500b4700 2024-03-25T18:05:23.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:23.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.876+0000 7f814d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:23.930 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.930+0000 7ff0bae0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:05:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.949+0000 7f815704a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b", "id": 4}] 2024-03-25T18:05:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.949+0000 7f815704a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b", "id": 4}] 2024-03-25T18:05:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.949+0000 7f815704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.949+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd new", "uuid": "1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b", "id": 4} v 0) v1 -- 0x7f8150069f30 con 0x7f81500b4700 2024-03-25T18:05:23.976 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:23.976+0000 7ff0bae0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:05:23.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.976+0000 7f814d7fa640 1 -- 172.21.15.157:0/1921224179 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 626530196 0 1697012583) 0x7f813c005490 con 0x7f81500b4700 2024-03-25T18:05:23.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.976+0000 7f814d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b", "id": 4}] 2024-03-25T18:05:23.977 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.976+0000 7f814d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:23.977 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-03-25T18:05:23.978 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 10 librados: watch_flush enter 2024-03-25T18:05:23.978 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 10 librados: watch_flush exit 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 10 client.4145.objecter successfully canceled tick 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 10 monclient: shutdown 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 >> v1:172.21.15.84:6790/0 conn(0x7f81500b4700 legacy=0x7f8150138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 shutdown_connections 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 >> 172.21.15.157:0/1921224179 conn(0x7f81500668e0 msgr2=0x7f81500b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 shutdown_connections 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 -- 172.21.15.157:0/1921224179 wait complete. 2024-03-25T18:05:23.979 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:23.977+0000 7f815704a640 1 librados: shutdown 2024-03-25T18:05:23.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new dbf339f4-d0d7-4c48-b9d9-5dc391839d12 5 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.087+0000 7f0346c7b640 10 monclient: get_monmap_and_config 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.087+0000 7f0346c7b640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.087+0000 7f0346c7b640 10 monclient: monmap: 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:24.088578+0000 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:24.088578+0000 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.088 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.089 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.089 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:24.089 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:24.089 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 1 Processor -- start 2024-03-25T18:05:24.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 1 -- start start 2024-03-25T18:05:24.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient: init 2024-03-25T18:05:24.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f03400b4700 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f0340142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f03400a96b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.088+0000 7f0346c7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400a9d60 con 0x7f03400b4700 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0346c7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034013ff10 con 0x7f03400a96b0 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0346c7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400aa1f0 con 0x7f0340142a20 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0346c7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0346c7b640 10 monclient(hunting): authenticate will time out at 6924.354702s 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0345218640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f03400a96b0 0x7f034013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42296/0 (socket says 172.21.15.157:42296) 2024-03-25T18:05:24.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.089+0000 7f0345218640 1 -- 172.21.15.157:0/1740454778 learned_addr learned my addr 172.21.15.157:0/1740454778 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.092 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898760513 0 0) 0x7f034013ff10 con 0x7f03400a96b0 2024-03-25T18:05:24.092 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.091+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0330003520 con 0x7f03400a96b0 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226171801 0 0) 0x7f03400aa1f0 con 0x7f0340142a20 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034013ff10 con 0x7f0340142a20 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223605843 0 0) 0x7f03400a9d60 con 0x7f03400b4700 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.093 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.094 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:05:24.094 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.092+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03400aa1f0 con 0x7f03400b4700 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061482072 0 0) 0x7f0330003520 con 0x7f03400a96b0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 >> v1:172.21.15.84:6790/0 conn(0x7f0340142a20 legacy=0x7f034013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 >> v1:172.21.15.84:6789/0 conn(0x7f03400b4700 legacy=0x7f0340142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400b4ae0 con 0x7f03400a96b0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0338001a10 con 0x7f03400a96b0 2024-03-25T18:05:24.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f0346c7b640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f0346c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f033f7fe640 10 monclient: dump: 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.097 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.098 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.098 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.095+0000 7f0346c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0338001b70 con 0x7f03400a96b0 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 1 -- 172.21.15.157:0/1740454778 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0338001f80 con 0x7f03400a96b0 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f0346c7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f0346c7b640 4 set_mon_vals no callback set 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.096+0000 7f033f7fe640 10 monclient: dump: 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.101 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 10 monclient: shutdown 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 1 -- 172.21.15.157:0/1740454778 >> v1:172.21.15.157:6789/0 conn(0x7f03400a96b0 legacy=0x7f034013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 1 -- 172.21.15.157:0/1740454778 shutdown_connections 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 1 -- 172.21.15.157:0/1740454778 >> 172.21.15.157:0/1740454778 conn(0x7f03400668e0 msgr2=0x7f0340066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 1 -- 172.21.15.157:0/1740454778 shutdown_connections 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.097+0000 7f0346c7b640 1 -- 172.21.15.157:0/1740454778 wait complete. 2024-03-25T18:05:24.102 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 10 monclient: monmap: 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 librados: starting msgr at 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 librados: starting objecter 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 Processor -- start 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 -- start start 2024-03-25T18:05:24.103 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 librados: setting wanted keys 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 1 librados: calling monclient init 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.098+0000 7f0346c7b640 10 monclient: init 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f03400a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f03400b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f0340142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400af020 con 0x7f03400a96b0 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400ae210 con 0x7f03400b4700 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400ae4f0 con 0x7f0340142a20 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f0346c7b640 10 monclient(hunting): authenticate will time out at 6924.364999s 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f033ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f03400b4700 0x7f03400acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42304/0 (socket says 172.21.15.157:42304) 2024-03-25T18:05:24.104 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f033ffff640 1 -- 172.21.15.157:0/2066942703 learned_addr learned my addr 172.21.15.157:0/2066942703 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f033d7fa640 10 client.?.objecter ms_handle_connect 0x7f03400b4700 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.099+0000 7f033d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.100+0000 7f033d7fa640 10 client.?.objecter ms_handle_connect 0x7f03400a96b0 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.100+0000 7f033d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.100+0000 7f033d7fa640 10 client.?.objecter ms_handle_connect 0x7f0340142a20 2024-03-25T18:05:24.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.100+0000 7f033d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.106 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731478299 0 0) 0x7f03400ae210 con 0x7f03400b4700 2024-03-25T18:05:24.106 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.106 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.101+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324004eb0 con 0x7f03400b4700 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610281292 0 0) 0x7f03400ae4f0 con 0x7f0340142a20 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03400ae210 con 0x7f0340142a20 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228408957 0 0) 0x7f03400af020 con 0x7f03400a96b0 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.107 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.108 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:05:24.108 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.102+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03400ae4f0 con 0x7f03400a96b0 2024-03-25T18:05:24.108 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.106+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187508134 0 0) 0x7f0324004eb0 con 0x7f03400b4700 2024-03-25T18:05:24.108 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.106+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03400af020 con 0x7f03400b4700 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f032c001ad0 con 0x7f03400b4700 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): dump: 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.109 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4047819335 0 0) 0x7f03400ae210 con 0x7f0340142a20 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324004eb0 con 0x7f0340142a20 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0338001e80 con 0x7f0340142a20 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): dump: 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.110 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1526221466 0 0) 0x7f03400ae4f0 con 0x7f03400a96b0 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03400ae210 con 0x7f03400a96b0 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0334001830 con 0x7f03400a96b0 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.107+0000 7f033d7fa640 10 monclient(hunting): dump: 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.111 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.112 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1131462621 0 0) 0x7f03400af020 con 0x7f03400b4700 2024-03-25T18:05:24.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 >> v1:172.21.15.84:6790/0 conn(0x7f0340142a20 legacy=0x7f03401e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 >> v1:172.21.15.84:6789/0 conn(0x7f03400a96b0 legacy=0x7f03400ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 1 monclient: found mon.b 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400ac0e0 con 0x7f03400b4700 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.109+0000 7f033d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:05:24.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _renew_subs 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03400ad340 con 0x7f03400b4700 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _renew_subs 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f034013b070 con 0x7f03400b4700 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 1 librados: init done 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.110+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034013b370 con 0x7f03400b4700 2024-03-25T18:05:24.139 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:24.139+0000 7ff0bae0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f032c001c30 con 0x7f03400b4700 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f032c003a60 con 0x7f03400b4700 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: dump: 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.209 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f032c003fd0 con 0x7f03400b4700 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033effd640 4 set_mon_vals no callback set 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 814375173 0 0) 0x7f032c0067a0 con 0x7f03400b4700 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 client.4147.objecter ms_dispatch 0x7f0340000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:05:24.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f032c006bc0 con 0x7f03400b4700 2024-03-25T18:05:24.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.208+0000 7f033d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:24.280 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.279+0000 7f0346c7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dbf339f4-d0d7-4c48-b9d9-5dc391839d12", "id": 5}] 2024-03-25T18:05:24.280 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.279+0000 7f0346c7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dbf339f4-d0d7-4c48-b9d9-5dc391839d12", "id": 5}] 2024-03-25T18:05:24.280 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.279+0000 7f0346c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.280 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.279+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dbf339f4-d0d7-4c48-b9d9-5dc391839d12", "id": 5} v 0) v1 -- 0x7f034004f730 con 0x7f03400b4700 2024-03-25T18:05:24.343 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.342+0000 7f033d7fa640 1 -- 172.21.15.157:0/2066942703 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dbf339f4-d0d7-4c48-b9d9-5dc391839d12", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1785476348 0 1988412176) 0x7f032c004460 con 0x7f03400b4700 2024-03-25T18:05:24.343 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.342+0000 7f033d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dbf339f4-d0d7-4c48-b9d9-5dc391839d12", "id": 5}] 2024-03-25T18:05:24.343 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.342+0000 7f033d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:24.343 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-03-25T18:05:24.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 10 librados: watch_flush enter 2024-03-25T18:05:24.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 10 librados: watch_flush exit 2024-03-25T18:05:24.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:05:24.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 10 monclient: shutdown 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 >> v1:172.21.15.157:6789/0 conn(0x7f03400b4700 legacy=0x7f03400acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 shutdown_connections 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 >> 172.21.15.157:0/2066942703 conn(0x7f03400668e0 msgr2=0x7f03400b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 shutdown_connections 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 -- 172.21.15.157:0/2066942703 wait complete. 2024-03-25T18:05:24.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.343+0000 7f0346c7b640 1 librados: shutdown 2024-03-25T18:05:24.354 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 7f268061-9caa-483c-b5d3-b7f7b83b9b45 6 2024-03-25T18:05:24.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.452+0000 7f2c2fa6a640 10 monclient: get_monmap_and_config 2024-03-25T18:05:24.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.452+0000 7f2c2fa6a640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.452+0000 7f2c2fa6a640 10 monclient: monmap: 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:24.453409+0000 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:24.453409+0000 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:24.453 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 1 Processor -- start 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 1 -- start start 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient: init 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-c con 0x7f2c280706d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-a con 0x7f2c28070ab0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-b con 0x7f2c28071930 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28072a00 con 0x7f2c28070ab0 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28140590 con 0x7f2c280706d0 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2813a520 con 0x7f2c28071930 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.453+0000 7f2c2fa6a640 10 monclient(hunting): authenticate will time out at 6924.719575s 2024-03-25T18:05:24.456 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.454+0000 7f2c2d806640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2c280706d0 0x7f2c280b0840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42306/0 (socket says 172.21.15.157:42306) 2024-03-25T18:05:24.456 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.454+0000 7f2c2d806640 1 -- 172.21.15.157:0/2478265287 learned_addr learned my addr 172.21.15.157:0/2478265287 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121801510 0 0) 0x7f2c28140590 con 0x7f2c280706d0 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c14003520 con 0x7f2c280706d0 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069781592 0 0) 0x7f2c2813a520 con 0x7f2c28071930 2024-03-25T18:05:24.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.456+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c28140590 con 0x7f2c28071930 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054447025 0 0) 0x7f2c28072a00 con 0x7f2c28070ab0 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.458 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.459 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.459 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:05:24.459 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.457+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2813a520 con 0x7f2c28070ab0 2024-03-25T18:05:24.460 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287308370 0 0) 0x7f2c14003520 con 0x7f2c280706d0 2024-03-25T18:05:24.460 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 >> v1:172.21.15.84:6790/0 conn(0x7f2c28071930 legacy=0x7f2c28139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 >> v1:172.21.15.84:6789/0 conn(0x7f2c28070ab0 legacy=0x7f2c28140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 monclient: found mon.noname-c 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28072c70 con 0x7f2c280706d0 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2c24001690 con 0x7f2c280706d0 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2c804640 10 monclient: dump: 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.461 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2fa6a640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:05:24.462 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.460+0000 7f2c2fa6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:24.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c240017f0 con 0x7f2c280706d0 2024-03-25T18:05:24.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:24.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 1 -- 172.21.15.157:0/2478265287 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2c24001b80 con 0x7f2c280706d0 2024-03-25T18:05:24.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2c804640 10 monclient: dump: 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2fa6a640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:24.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.461+0000 7f2c2fa6a640 4 set_mon_vals no callback set 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 10 monclient: shutdown 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2478265287 >> v1:172.21.15.157:6789/0 conn(0x7f2c280706d0 legacy=0x7f2c280b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2478265287 shutdown_connections 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2478265287 >> 172.21.15.157:0/2478265287 conn(0x7f2c280b5e40 msgr2=0x7f2c280b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2478265287 shutdown_connections 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.462+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2478265287 wait complete. 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient: monmap: 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.467 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 librados: starting msgr at 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 librados: starting objecter 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 Processor -- start 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 -- start start 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 librados: setting wanted keys 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 librados: calling monclient init 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient: init 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-b con 0x7f2c280706d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-c con 0x7f2c28070ab0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.468 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient(hunting): picked mon.noname-a con 0x7f2c28071930 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2806d1f0 con 0x7f2c28071930 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2812d930 con 0x7f2c280706d0 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28141b50 con 0x7f2c28070ab0 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.463+0000 7f2c2fa6a640 10 monclient(hunting): authenticate will time out at 6924.729631s 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c2d806640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2c280706d0 0x7f2c281daaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42308/0 (socket says 172.21.15.157:42308) 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c2d806640 1 -- 172.21.15.157:0/2485048433 learned_addr learned my addr 172.21.15.157:0/2485048433 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2c280706d0 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2c28070ab0 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2c28071930 2024-03-25T18:05:24.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.464+0000 7f2c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.470 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302704968 0 0) 0x7f2c2812d930 con 0x7f2c280706d0 2024-03-25T18:05:24.470 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.470 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.470 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.470 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.466+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c004eb0 con 0x7f2c280706d0 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063799617 0 0) 0x7f2c28141b50 con 0x7f2c28070ab0 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2812d930 con 0x7f2c28070ab0 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071169640 0 0) 0x7f2c2806d1f0 con 0x7f2c28071930 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:05:24.471 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.467+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c28141b50 con 0x7f2c28071930 2024-03-25T18:05:24.472 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146907330 0 0) 0x7f2c2812d930 con 0x7f2c28070ab0 2024-03-25T18:05:24.472 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c2806d1f0 con 0x7f2c28070ab0 2024-03-25T18:05:24.472 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2c18001870 con 0x7f2c28070ab0 2024-03-25T18:05:24.472 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541199580 0 0) 0x7f2c0c004eb0 con 0x7f2c280706d0 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c2812d930 con 0x7f2c280706d0 2024-03-25T18:05:24.473 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2c24001b00 con 0x7f2c280706d0 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.470+0000 7f2c1e7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.471+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4288939479 0 0) 0x7f2c28141b50 con 0x7f2c28071930 2024-03-25T18:05:24.474 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.471+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c0c004eb0 con 0x7f2c28071930 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2212803452 0 0) 0x7f2c2812d930 con 0x7f2c280706d0 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 >> v1:172.21.15.84:6790/0 conn(0x7f2c28070ab0 legacy=0x7f2c2812cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 >> v1:172.21.15.84:6789/0 conn(0x7f2c28071930 legacy=0x7f2c2812d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 1 monclient: found mon.b 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2806c1e0 con 0x7f2c280706d0 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 10 monclient: _renew_subs 2024-03-25T18:05:24.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2806c430 con 0x7f2c280706d0 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 10 monclient: _renew_subs 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2806cdd0 con 0x7f2c280706d0 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.474+0000 7f2c2fa6a640 1 librados: init done 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.475+0000 7f2c2fa6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.475+0000 7f2c2fa6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.475+0000 7f2c2fa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.477 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.475+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2804f730 con 0x7f2c280706d0 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c24004a90 con 0x7f2c280706d0 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2c24004da0 con 0x7f2c280706d0 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: dump: 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.574 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2c24005290 con 0x7f2c280706d0 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2995667158 0 0) 0x7f2c240068a0 con 0x7f2c280706d0 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 client.4153.objecter ms_dispatch 0x7f2c28000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1ffff640 4 set_mon_vals no callback set 2024-03-25T18:05:24.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:05:24.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2c24018dc0 con 0x7f2c280706d0 2024-03-25T18:05:24.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.573+0000 7f2c1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:24.645 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.645+0000 7f2c2fa6a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7f268061-9caa-483c-b5d3-b7f7b83b9b45", "id": 6}] 2024-03-25T18:05:24.645 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.645+0000 7f2c2fa6a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7f268061-9caa-483c-b5d3-b7f7b83b9b45", "id": 6}] 2024-03-25T18:05:24.645 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.645+0000 7f2c2fa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.645 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.645+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7f268061-9caa-483c-b5d3-b7f7b83b9b45", "id": 6} v 0) v1 -- 0x7f2c28069f30 con 0x7f2c280706d0 2024-03-25T18:05:24.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.705+0000 7f2c1e7fc640 1 -- 172.21.15.157:0/2485048433 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f268061-9caa-483c-b5d3-b7f7b83b9b45", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2174544125 0 1113811849) 0x7f2c240062b0 con 0x7f2c280706d0 2024-03-25T18:05:24.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.705+0000 7f2c1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7f268061-9caa-483c-b5d3-b7f7b83b9b45", "id": 6}] 2024-03-25T18:05:24.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.705+0000 7f2c1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:24.706 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-03-25T18:05:24.707 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 10 librados: watch_flush enter 2024-03-25T18:05:24.707 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 10 librados: watch_flush exit 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 10 monclient: shutdown 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.706+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 >> v1:172.21.15.157:6789/0 conn(0x7f2c280706d0 legacy=0x7f2c281daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.707+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 shutdown_connections 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.707+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 >> 172.21.15.157:0/2485048433 conn(0x7f2c280b5e40 msgr2=0x7f2c28133190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.707+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 shutdown_connections 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.707+0000 7f2c2fa6a640 1 -- 172.21.15.157:0/2485048433 wait complete. 2024-03-25T18:05:24.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.707+0000 7f2c2fa6a640 1 librados: shutdown 2024-03-25T18:05:24.718 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new c150a5c7-47a4-49d5-aa05-058475f3559c 7 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.818+0000 7f4b5c214640 10 monclient: get_monmap_and_config 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.818+0000 7f4b5c214640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.818+0000 7f4b5c214640 10 monclient: monmap: 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:05:24.819679+0000 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:05:24.819679+0000 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.819 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:24.820 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:24.820 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.819+0000 7f4b5c214640 1 Processor -- start 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.819+0000 7f4b5c214640 1 -- start start 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.819+0000 7f4b5c214640 10 monclient: init 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-c con 0x7f4b540b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-a con 0x7f4b54142a20 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-b con 0x7f4b540a96b0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b540a9d60 con 0x7f4b54142a20 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5413ff10 con 0x7f4b540b4700 2024-03-25T18:05:24.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b540aa1f0 con 0x7f4b540a96b0 2024-03-25T18:05:24.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.820+0000 7f4b5c214640 10 monclient(hunting): authenticate will time out at 6925.086263s 2024-03-25T18:05:24.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.821+0000 7f4b59fb0640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4b540b4700 0x7f4b54142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42324/0 (socket says 172.21.15.157:42324) 2024-03-25T18:05:24.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.821+0000 7f4b59fb0640 1 -- 172.21.15.157:0/4225621685 learned_addr learned my addr 172.21.15.157:0/4225621685 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.823 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261762595 0 0) 0x7f4b5413ff10 con 0x7f4b540b4700 2024-03-25T18:05:24.823 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.824 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.824 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.824 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:05:24.824 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.823+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b40003520 con 0x7f4b540b4700 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730555846 0 0) 0x7f4b540aa1f0 con 0x7f4b540a96b0 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:05:24.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5413ff10 con 0x7f4b540a96b0 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674973026 0 0) 0x7f4b540a9d60 con 0x7f4b54142a20 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:05:24.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.825+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b540aa1f0 con 0x7f4b54142a20 2024-03-25T18:05:24.827 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589505119 0 0) 0x7f4b40003520 con 0x7f4b540b4700 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 >> v1:172.21.15.84:6790/0 conn(0x7f4b540a96b0 legacy=0x7f4b5413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 >> v1:172.21.15.84:6789/0 conn(0x7f4b54142a20 legacy=0x7f4b5413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 monclient: found mon.noname-c 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b540b4ae0 con 0x7f4b540b4700 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b50001690 con 0x7f4b540b4700 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b5c214640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.828 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b58fae640 10 monclient: dump: 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.829 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.826+0000 7f4b5c214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b500017f0 con 0x7f4b540b4700 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 1 -- 172.21.15.157:0/4225621685 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b50001c00 con 0x7f4b540b4700 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b5c214640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b5c214640 4 set_mon_vals no callback set 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b58fae640 10 monclient: dump: 2024-03-25T18:05:24.831 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b5c214640 10 monclient: shutdown 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b5c214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.828+0000 7f4b5c214640 1 -- 172.21.15.157:0/4225621685 >> v1:172.21.15.157:6789/0 conn(0x7f4b540b4700 legacy=0x7f4b54142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.829+0000 7f4b5c214640 1 -- 172.21.15.157:0/4225621685 shutdown_connections 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.829+0000 7f4b5c214640 1 -- 172.21.15.157:0/4225621685 >> 172.21.15.157:0/4225621685 conn(0x7f4b540668e0 msgr2=0x7f4b54066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:24.832 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.829+0000 7f4b5c214640 1 -- 172.21.15.157:0/4225621685 shutdown_connections 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.829+0000 7f4b5c214640 1 -- 172.21.15.157:0/4225621685 wait complete. 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 10 monclient: build_initial_monmap 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 10 monclient: monmap: 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 librados: starting msgr at 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 librados: starting objecter 2024-03-25T18:05:24.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 Processor -- start 2024-03-25T18:05:24.834 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 -- start start 2024-03-25T18:05:24.834 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 librados: setting wanted keys 2024-03-25T18:05:24.834 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 1 librados: calling monclient init 2024-03-25T18:05:24.834 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.830+0000 7f4b5c214640 10 monclient: init 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-a con 0x7f4b540a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-b con 0x7f4b540b4700 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient(hunting): picked mon.noname-c con 0x7f4b54142a20 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5414eca0 con 0x7f4b540a96b0 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b541d74d0 con 0x7f4b540b4700 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b541d8350 con 0x7f4b54142a20 2024-03-25T18:05:24.836 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.831+0000 7f4b5c214640 10 monclient(hunting): authenticate will time out at 6925.097346s 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b597af640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4b540b4700 0x7f4b540acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42332/0 (socket says 172.21.15.157:42332) 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b597af640 1 -- 172.21.15.157:0/4170691878 learned_addr learned my addr 172.21.15.157:0/4170691878 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter ms_handle_connect 0x7f4b540b4700 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter ms_handle_connect 0x7f4b540a96b0 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter ms_handle_connect 0x7f4b54142a20 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.832+0000 7f4b4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547614145 0 0) 0x7f4b541d74d0 con 0x7f4b540b4700 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.837 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b38004eb0 con 0x7f4b540b4700 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378114712 0 0) 0x7f4b540b4ae0 con 0x7f4b540a96b0 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b541d74d0 con 0x7f4b540a96b0 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611396605 0 0) 0x7f4b541d8350 con 0x7f4b54142a20 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:05:24.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.834+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b540b4ae0 con 0x7f4b54142a20 2024-03-25T18:05:24.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615348679 0 0) 0x7f4b38004eb0 con 0x7f4b540b4700 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b541d8350 con 0x7f4b540b4700 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b44001a10 con 0x7f4b540b4700 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.840 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.838+0000 7f4b4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902460119 0 0) 0x7f4b541d74d0 con 0x7f4b540a96b0 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b38004eb0 con 0x7f4b540a96b0 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b50001960 con 0x7f4b540a96b0 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.839+0000 7f4b4a7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.841 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.842 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.842 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2029744324 0 0) 0x7f4b540b4ae0 con 0x7f4b54142a20 2024-03-25T18:05:24.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b541d74d0 con 0x7f4b54142a20 2024-03-25T18:05:24.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b4c001bd0 con 0x7f4b54142a20 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.840+0000 7f4b4a7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:24.844 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3223447593 0 0) 0x7f4b38004eb0 con 0x7f4b540a96b0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 >> v1:172.21.15.84:6790/0 conn(0x7f4b54142a20 legacy=0x7f4b541d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 >> v1:172.21.15.157:6789/0 conn(0x7f4b540b4700 legacy=0x7f4b540acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 1 monclient: found mon.a 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b540abd90 con 0x7f4b540a96b0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 10 monclient: _renew_subs 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:24.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b541d7720 con 0x7f4b540a96b0 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 10 monclient: _renew_subs 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b540ad240 con 0x7f4b540a96b0 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.842+0000 7f4b5c214640 1 librados: init done 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.843+0000 7f4b5c214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.843+0000 7f4b5c214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.843+0000 7f4b5c214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:24.846 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:24.843+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5404f730 con 0x7f4b540a96b0 2024-03-25T18:05:24.958 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:24.957+0000 7fdc2ee3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:05:24.976 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:24.976+0000 7ff0bae0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b500047e0 con 0x7f4b540a96b0 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4b50004af0 con 0x7f4b540a96b0 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 10 monclient: dump: 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-25T18:05:25.023 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 3 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4b50004fe0 con 0x7f4b540a96b0 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4bfff640 4 set_mon_vals no callback set 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2848587995 0 0) 0x7f4b500066b0 con 0x7f4b540a96b0 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 10 client.4158.objecter ms_dispatch 0x7f4b54000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:05:25.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.022+0000 7f4b4a7fc640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-25T18:05:25.025 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.025+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4b50006160 con 0x7f4b540a96b0 2024-03-25T18:05:25.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.025+0000 7f4b4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:25.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.025+0000 7f4b4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:25.096 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.096+0000 7f4b5c214640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c150a5c7-47a4-49d5-aa05-058475f3559c", "id": 7}] 2024-03-25T18:05:25.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.096+0000 7f4b5c214640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c150a5c7-47a4-49d5-aa05-058475f3559c", "id": 7}] 2024-03-25T18:05:25.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.096+0000 7f4b5c214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:25.097 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.096+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c150a5c7-47a4-49d5-aa05-058475f3559c", "id": 7} v 0) v1 -- 0x7f4b540b7470 con 0x7f4b540a96b0 2024-03-25T18:05:25.222 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.221+0000 7f4b4a7fc640 1 -- 172.21.15.157:0/4170691878 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c150a5c7-47a4-49d5-aa05-058475f3559c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 97745620 0 1371665406) 0x7f4b50006410 con 0x7f4b540a96b0 2024-03-25T18:05:25.222 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.221+0000 7f4b4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c150a5c7-47a4-49d5-aa05-058475f3559c", "id": 7}] 2024-03-25T18:05:25.222 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.221+0000 7f4b4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:25.222 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-03-25T18:05:25.223 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 10 librados: watch_flush enter 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 10 librados: watch_flush exit 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 10 client.4158.objecter successfully canceled tick 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 10 monclient: shutdown 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 >> v1:172.21.15.84:6789/0 conn(0x7f4b540a96b0 legacy=0x7f4b541d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 shutdown_connections 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 >> 172.21.15.157:0/4170691878 conn(0x7f4b540668e0 msgr2=0x7f4b540b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 shutdown_connections 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 -- 172.21.15.157:0/4170691878 wait complete. 2024-03-25T18:05:25.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-25T18:05:25.222+0000 7f4b5c214640 1 librados: shutdown 2024-03-25T18:05:25.233 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.233 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:05:25.233 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:05:25.238 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:05:25.238 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.238 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:05:25.238 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:05:25.239 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:25.237+0000 7fdc2ee3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:05:25.240 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:05:25.240 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.241 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:05:25.241 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:05:25.243 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:05:25.243 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.243 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:05:25.243 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:05:25.245 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:05:25.245 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.245 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:05:25.245 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:05:25.247 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:05:25.248 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.248 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:05:25.248 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:05:25.250 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:05:25.250 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.250 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:05:25.250 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:05:25.252 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:05:25.252 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:05:25.252 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:05:25.252 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:05:25.254 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:05:25.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:25.271 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:25.271+0000 7ff0bae0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:05:25.301 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:25.300+0000 7fdc2ee3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:05:25.350 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:25.350+0000 7ff0bae0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.406+0000 7fe273cb3640 10 monclient: get_monmap_and_config 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.406+0000 7fe273cb3640 10 monclient: build_initial_monmap 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.406+0000 7fe273cb3640 10 monclient: monmap: 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:05:25.407013+0000 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:05:25.407013+0000 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:25.407 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:25.408 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.407+0000 7fe273cb3640 1 Processor -- start 2024-03-25T18:05:25.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.407+0000 7fe273cb3640 1 -- start start 2024-03-25T18:05:25.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.407+0000 7fe273cb3640 10 monclient: init 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-b con 0x7fe26c0b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-c con 0x7fe26c142a20 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c0a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0a9d60 con 0x7fe26c0a96b0 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c13ff10 con 0x7fe26c142a20 2024-03-25T18:05:25.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0aa1f0 con 0x7fe26c0b4700 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe273cb3640 10 monclient(hunting): authenticate will time out at 6969.706581s 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe271a4f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe26c0b4700 0x7fe26c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38924/0 (socket says 172.21.15.84:38924) 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.408+0000 7fe271a4f640 1 -- 172.21.15.84:0/2835651840 learned_addr learned my addr 172.21.15.84:0/2835651840 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713132318 0 0) 0x7fe26c13ff10 con 0x7fe26c142a20 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.412+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe258003520 con 0x7fe26c142a20 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580855342 0 0) 0x7fe26c0aa1f0 con 0x7fe26c0b4700 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:05:25.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.413+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c13ff10 con 0x7fe26c0b4700 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180407469 0 0) 0x7fe26c0a9d60 con 0x7fe26c0a96b0 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:05:25.416 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.416+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c0aa1f0 con 0x7fe26c0a96b0 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.417+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712795148 0 0) 0x7fe258003520 con 0x7fe26c142a20 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.417+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 >> v1:172.21.15.84:6790/0 conn(0x7fe26c0b4700 legacy=0x7fe26c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.417+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 >> v1:172.21.15.84:6789/0 conn(0x7fe26c0a96b0 legacy=0x7fe26c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: _finish_hunting 0 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 1 monclient: found mon.noname-c 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0b4ae0 con 0x7fe26c142a20 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: _finish_auth 0 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: _check_auth_tickets 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe25c001690 con 0x7fe26c142a20 2024-03-25T18:05:25.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe273cb3640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe273cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe270a4d640 10 monclient: dump: 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.418+0000 7fe273cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe25c0018b0 con 0x7fe26c142a20 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 1 -- 172.21.15.84:0/2835651840 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe25c001c40 con 0x7fe26c142a20 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe270a4d640 10 monclient: dump: 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe273cb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe273cb3640 4 set_mon_vals no callback set 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe273cb3640 10 monclient: shutdown 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe273cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.421+0000 7fe273cb3640 1 -- 172.21.15.84:0/2835651840 >> v1:172.21.15.157:6789/0 conn(0x7fe26c142a20 legacy=0x7fe26c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.422+0000 7fe273cb3640 1 -- 172.21.15.84:0/2835651840 shutdown_connections 2024-03-25T18:05:25.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.422+0000 7fe273cb3640 1 -- 172.21.15.84:0/2835651840 >> 172.21.15.84:0/2835651840 conn(0x7fe26c0668e0 msgr2=0x7fe26c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.422+0000 7fe273cb3640 1 -- 172.21.15.84:0/2835651840 shutdown_connections 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.422+0000 7fe273cb3640 1 -- 172.21.15.84:0/2835651840 wait complete. 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 10 monclient: build_initial_monmap 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 10 monclient: monmap: 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 librados: starting msgr at 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 librados: starting objecter 2024-03-25T18:05:25.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 Processor -- start 2024-03-25T18:05:25.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 -- start start 2024-03-25T18:05:25.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 librados: setting wanted keys 2024-03-25T18:05:25.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 1 librados: calling monclient init 2024-03-25T18:05:25.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.423+0000 7fe273cb3640 10 monclient: init 2024-03-25T18:05:25.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:25.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c0a96b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-c con 0x7fe26c0b4700 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient(hunting): picked mon.noname-b con 0x7fe26c142a20 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0ade40 con 0x7fe26c0a96b0 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0ae0b0 con 0x7fe26c142a20 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0ae390 con 0x7fe26c0b4700 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe273cb3640 10 monclient(hunting): authenticate will time out at 6969.722243s 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe272250640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe26c142a20 0x7fe26c1e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:57298/0 (socket says 172.21.15.84:57298) 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.424+0000 7fe272250640 1 -- 172.21.15.84:0/1995705614 learned_addr learned my addr 172.21.15.84:0/1995705614 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter ms_handle_connect 0x7fe26c142a20 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter ms_handle_connect 0x7fe26c0b4700 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter ms_handle_connect 0x7fe26c0a96b0 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.425+0000 7fe2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764347616 0 0) 0x7fe26c0ae390 con 0x7fe26c0b4700 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.427+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe250004eb0 con 0x7fe26c0b4700 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944129845 0 0) 0x7fe26c0ae0b0 con 0x7fe26c142a20 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c0ae390 con 0x7fe26c142a20 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.428+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636202648 0 0) 0x7fe26c0ade40 con 0x7fe26c0a96b0 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.429+0000 7fe2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.429+0000 7fe2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.429+0000 7fe2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.429+0000 7fe2627fc640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:05:25.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.429+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c0ae0b0 con 0x7fe26c0a96b0 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542796430 0 0) 0x7fe250004eb0 con 0x7fe26c0b4700 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe26c0ade40 con 0x7fe26c0b4700 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe25c0015d0 con 0x7fe26c0b4700 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 10 monclient(hunting): dump: 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.433 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.433+0000 7fe2627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765648645 0 0) 0x7fe26c0ae0b0 con 0x7fe26c0a96b0 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe250004eb0 con 0x7fe26c0a96b0 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2680018f0 con 0x7fe26c0a96b0 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): dump: 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.436 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798570724 0 0) 0x7fe26c0ae390 con 0x7fe26c142a20 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe26c0ae0b0 con 0x7fe26c142a20 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe264001ce0 con 0x7fe26c142a20 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient(hunting): dump: 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.438 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.438 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.438 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 592699027 0 0) 0x7fe26c0ade40 con 0x7fe26c0b4700 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 >> v1:172.21.15.157:6789/0 conn(0x7fe26c142a20 legacy=0x7fe26c1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 >> v1:172.21.15.84:6789/0 conn(0x7fe26c0a96b0 legacy=0x7fe26c13aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 monclient: found mon.c 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0ad060 con 0x7fe26c0b4700 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:25.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe2627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe273cb3640 5 monclient: authenticate success, global_id 4169 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe273cb3640 10 monclient: _renew_subs 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe273cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.436+0000 7fe273cb3640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c0abcc0 con 0x7fe26c0b4700 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.437+0000 7fe273cb3640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.437+0000 7fe273cb3640 10 monclient: _renew_subs 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.437+0000 7fe273cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.437+0000 7fe273cb3640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c0ac960 con 0x7fe26c0b4700 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.437+0000 7fe273cb3640 1 librados: init done 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.438+0000 7fe257fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.438+0000 7fe257fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.439+0000 7fe257fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.439+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c04f730 con 0x7fe26c0b4700 2024-03-25T18:05:25.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.439+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe25c0038a0 con 0x7fe26c0b4700 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.439+0000 7fe2627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.439+0000 7fe263fff640 4 set_mon_vals no callback set 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.440+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe25c003bb0 con 0x7fe26c0b4700 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.440+0000 7fe2627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.440+0000 7fe2627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.440+0000 7fe2627fc640 10 monclient: dump: 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:25.444 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:05:25.500 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:25.500+0000 7fdc2ee3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe25c0040a0 con 0x7fe26c0b4700 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fe25c0058b0 con 0x7fe26c0b4700 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 10 client.4169.objecter ms_dispatch 0x7fe26c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 20 client.4169.objecter dump_active .. 0 homeless 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe25c005bc0 con 0x7fe26c0b4700 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:25.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.538+0000 7fe2627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:25.556 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-25T18:05:25.556+0000 7ff0bae0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:05:25.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.661+0000 7fe257fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:25.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.661+0000 7fe257fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:25.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.661+0000 7fe257fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:25.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.661+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26c13e5d0 con 0x7fe26c0b4700 2024-03-25T18:05:25.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.770+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 112704272 0 0) 0x7fe25c0051e0 con 0x7fe26c0b4700 2024-03-25T18:05:25.836 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.836+0000 7fe2627fc640 1 -- 172.21.15.84:0/1995705614 <== mon.2 v1:172.21.15.84:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fe25c013180 con 0x7fe26c0b4700 2024-03-25T18:05:25.836 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.836+0000 7fe2627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:25.836 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.836+0000 7fe2627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:25.837 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-25T18:05:25.837 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.837+0000 7fe257fff640 10 librados: watch_flush enter 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.837+0000 7fe257fff640 10 librados: watch_flush exit 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 20 client.4169.objecter shutdown clearing up homeless session... 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 10 client.4169.objecter successfully canceled tick 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 10 monclient: shutdown 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 >> v1:172.21.15.84:6790/0 conn(0x7fe26c0b4700 legacy=0x7fe26c0ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:25.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 shutdown_connections 2024-03-25T18:05:25.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 >> 172.21.15.84:0/1995705614 conn(0x7fe26c0668e0 msgr2=0x7fe26c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:25.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 shutdown_connections 2024-03-25T18:05:25.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 -- 172.21.15.84:0/1995705614 wait complete. 2024-03-25T18:05:25.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-25T18:05:25.838+0000 7fe257fff640 1 librados: shutdown 2024-03-25T18:05:25.857 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:05:25.857 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:05:26.662 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:26.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.814+0000 7f8182b0e640 10 monclient: get_monmap_and_config 2024-03-25T18:05:26.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.814+0000 7f8182b0e640 10 monclient: build_initial_monmap 2024-03-25T18:05:26.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.814+0000 7f8182b0e640 10 monclient: monmap: 2024-03-25T18:05:26.815 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:26.815 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:05:26.814895+0000 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:05:26.814895+0000 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.815+0000 7f8182b0e640 1 Processor -- start 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.815+0000 7f8182b0e640 1 -- start start 2024-03-25T18:05:26.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.815+0000 7f8182b0e640 10 monclient: init 2024-03-25T18:05:26.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:26.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:26.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f817c0b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:26.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f817c0b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f817c14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0af940 con 0x7f817c0b1730 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0b0a70 con 0x7f817c14be90 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0b0d50 con 0x7f817c0b4d60 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.816+0000 7f8182b0e640 10 monclient(hunting): authenticate will time out at 6971.114845s 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.817+0000 7f817bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f817c0b1730 0x7f817c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42098/0 (socket says 172.21.15.84:42098) 2024-03-25T18:05:26.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:26.817+0000 7f817bfff640 1 -- 172.21.15.84:0/4112013662 learned_addr learned my addr 172.21.15.84:0/4112013662 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:27.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322688952 0 0) 0x7f817c0b0a70 con 0x7f817c14be90 2024-03-25T18:05:27.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:27.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:27.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:27.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 10 monclient(hunting): my global_id is 4180 2024-03-25T18:05:27.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.102+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c003520 con 0x7f817c14be90 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.112+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028899388 0 0) 0x7f817c0b0d50 con 0x7f817c0b4d60 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.112+0000 7f817b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.112+0000 7f817b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.112+0000 7f817b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.112+0000 7f817b7fe640 10 monclient(hunting): my global_id is 4184 2024-03-25T18:05:27.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.113+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c0b0a70 con 0x7f817c0b4d60 2024-03-25T18:05:27.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3658654833 0 0) 0x7f816c003520 con 0x7f817c14be90 2024-03-25T18:05:27.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 >> v1:172.21.15.84:6790/0 conn(0x7f817c0b4d60 legacy=0x7f817c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 >> v1:172.21.15.84:6789/0 conn(0x7f817c0b1730 legacy=0x7f817c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0b1030 con 0x7f817c14be90 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8174001a10 con 0x7f817c14be90 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:27.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f817b7fe640 10 monclient: dump: 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f8182b0e640 5 monclient: authenticate success, global_id 4180 2024-03-25T18:05:27.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.123+0000 7f8182b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:27.177 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:27.177+0000 7fdbcfb30640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8174001b70 con 0x7f817c14be90 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 1 -- 172.21.15.84:0/4112013662 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8174001f80 con 0x7f817c14be90 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f817b7fe640 10 monclient: dump: 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:27.178 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f8182b0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:27.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.177+0000 7f8182b0e640 4 set_mon_vals no callback set 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.178+0000 7f8182b0e640 10 monclient: shutdown 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.178+0000 7f8182b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.178+0000 7f8182b0e640 1 -- 172.21.15.84:0/4112013662 >> v1:172.21.15.157:6789/0 conn(0x7f817c14be90 legacy=0x7f817c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.178+0000 7f8182b0e640 1 -- 172.21.15.84:0/4112013662 shutdown_connections 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.178+0000 7f8182b0e640 1 -- 172.21.15.84:0/4112013662 >> 172.21.15.84:0/4112013662 conn(0x7f817c0617a0 msgr2=0x7f817c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.179+0000 7f8182b0e640 1 -- 172.21.15.84:0/4112013662 shutdown_connections 2024-03-25T18:05:27.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.179+0000 7f8182b0e640 1 -- 172.21.15.84:0/4112013662 wait complete. 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 10 monclient: build_initial_monmap 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 10 monclient: monmap: 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:27.183 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 librados: starting msgr at 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 librados: starting objecter 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 Processor -- start 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 -- start start 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 librados: setting wanted keys 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 1 librados: calling monclient init 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.180+0000 7f8182b0e640 10 monclient: init 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.181+0000 7f8182b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.181+0000 7f8182b0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f817c0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f817c0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f817c14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:27.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c13c5a0 con 0x7f817c0b4d60 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c1421b0 con 0x7f817c14be90 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c140840 con 0x7f817c0b1730 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.182+0000 7f8182b0e640 10 monclient(hunting): authenticate will time out at 6971.480301s 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.184+0000 7f81810ab640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f817c14be90 0x7f817c142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:57302/0 (socket says 172.21.15.84:57302) 2024-03-25T18:05:27.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.184+0000 7f81810ab640 1 -- 172.21.15.84:0/1410147727 learned_addr learned my addr 172.21.15.84:0/1410147727 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:27.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.187+0000 7f81797fa640 10 client.?.objecter ms_handle_connect 0x7f817c14be90 2024-03-25T18:05:27.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.187+0000 7f81797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:27.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36573317 0 0) 0x7f817c1421b0 con 0x7f817c14be90 2024-03-25T18:05:27.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:27.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:27.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:27.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 10 monclient(hunting): my global_id is 4183 2024-03-25T18:05:27.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.194+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8160004eb0 con 0x7f817c14be90 2024-03-25T18:05:27.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.199+0000 7f81797fa640 10 client.?.objecter ms_handle_connect 0x7f817c0b4d60 2024-03-25T18:05:27.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.199+0000 7f81797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:27.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886387611 0 0) 0x7f817c13c5a0 con 0x7f817c0b4d60 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 10 monclient(hunting): my global_id is 4182 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.202+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c1421b0 con 0x7f817c0b4d60 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927300392 0 0) 0x7f8160004eb0 con 0x7f817c14be90 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f817c13c5a0 con 0x7f817c14be90 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8174001f80 con 0x7f817c14be90 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:27.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 10 monclient(hunting): dump: 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:27.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.203+0000 7f81797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2193715834 0 0) 0x7f817c13c5a0 con 0x7f817c14be90 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 >> v1:172.21.15.84:6790/0 conn(0x7f817c0b1730 legacy=0x7f817c13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 >> v1:172.21.15.84:6789/0 conn(0x7f817c0b4d60 legacy=0x7f817c1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 10 monclient: _finish_hunting 0 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 1 monclient: found mon.b 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c1d7da0 con 0x7f817c14be90 2024-03-25T18:05:27.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 10 monclient: _finish_auth 0 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 10 monclient: _check_auth_tickets 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f81797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.209+0000 7f8182b0e640 5 monclient: authenticate success, global_id 4183 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 10 monclient: _renew_subs 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c141240 con 0x7f817c14be90 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 10 monclient: _renew_subs 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c140560 con 0x7f817c14be90 2024-03-25T18:05:27.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.210+0000 7f8182b0e640 1 librados: init done 2024-03-25T18:05:27.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.211+0000 7f8182b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:27.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.211+0000 7f8182b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:27.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.211+0000 7f8182b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.212+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c0afbd0 con 0x7f817c14be90 2024-03-25T18:05:27.219 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:05:27.219+0000 7fdbcfb30640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:05:27.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.427+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8174003c20 con 0x7f817c14be90 2024-03-25T18:05:27.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.427+0000 7f81797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:27.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.427+0000 7f817affd640 4 set_mon_vals no callback set 2024-03-25T18:05:27.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.619+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8174003ff0 con 0x7f817c14be90 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.619+0000 7f81797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.619+0000 7f81797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.619+0000 7f81797fa640 10 monclient: dump: 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:27.620 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:27.621 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:27.621 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:27.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.620+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1509705581 0 0) 0x7f81740041d0 con 0x7f817c14be90 2024-03-25T18:05:27.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.621+0000 7f81808aa640 10 monclient: get_auth_request con 0x7f816006dbc0 auth_method 0 2024-03-25T18:05:27.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.622+0000 7f81797fa640 10 client.4183.objecter ms_handle_connect 0x7f816006dbc0 2024-03-25T18:05:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.627+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f8174054ac0 con 0x7f817c14be90 2024-03-25T18:05:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.627+0000 7f81797fa640 10 client.4183.objecter ms_dispatch 0x7f817c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.627+0000 7f81797fa640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.627+0000 7f81797fa640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.627+0000 7f81797fa640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-25T18:05:27.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.630+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81740505d0 con 0x7f817c14be90 2024-03-25T18:05:27.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.630+0000 7f81797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:27.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.630+0000 7f81797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:27.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.969+0000 7f8182b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:27.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.969+0000 7f8182b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:27.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.969+0000 7f8182b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:27.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:27.969+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f817c0627b0 con 0x7f817c14be90 2024-03-25T18:05:28.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.035+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f8174050080 con 0x7f817c14be90 2024-03-25T18:05:28.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.142+0000 7f81797fa640 1 -- 172.21.15.84:0/1410147727 <== mon.1 v1:172.21.15.157:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f817407b440 con 0x7f817c14be90 2024-03-25T18:05:28.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.142+0000 7f81797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:28.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.142+0000 7f81797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:28.142 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:05:28.142 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:28.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 10 librados: watch_flush enter 2024-03-25T18:05:28.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 10 librados: watch_flush exit 2024-03-25T18:05:28.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-25T18:05:28.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 10 client.4183.objecter successfully canceled tick 2024-03-25T18:05:28.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 >> v1:172.21.15.157:6800/1801259026 conn(0x7f816006dbc0 legacy=0x7f8160070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 10 monclient: shutdown 2024-03-25T18:05:28.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:28.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.145+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 >> v1:172.21.15.157:6789/0 conn(0x7f817c14be90 legacy=0x7f817c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.146+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 shutdown_connections 2024-03-25T18:05:28.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.146+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 >> 172.21.15.84:0/1410147727 conn(0x7f817c0617a0 msgr2=0x7f817c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:28.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.146+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 shutdown_connections 2024-03-25T18:05:28.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.146+0000 7f8182b0e640 1 -- 172.21.15.84:0/1410147727 wait complete. 2024-03-25T18:05:28.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:28.146+0000 7f8182b0e640 1 librados: shutdown 2024-03-25T18:05:28.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:32.122 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:06.800 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.186 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:06.863 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.193 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:06.859 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.213 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:06.900 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.256 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:06.922 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.276 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:06.963 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.402 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:07.069 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.459 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:07.126 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.460 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:07.135 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.523 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:07.199 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.526 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:07.210 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.587 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:07.270 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.731 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-25T18:05:32.725+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:32.734 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:07.414 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.794 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:07.474 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.807 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-25T18:05:32.801+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:32.826 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-25T18:05:32.820+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:32.879 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:07.567 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:32.937 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:07.625 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.050 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-25T18:05:33.044+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:33.068 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-25T18:05:33.063+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:33.176 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-25T18:05:33.170+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:33.211 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:07.889 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.308 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:07.995 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.308 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:07.974 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.366 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-25T18:05:33.361+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:33.505 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-25T18:05:33.498+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:05:33.520 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:08.187 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.523 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:08.199 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.576 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:08.254 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.646 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:08.330 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.674 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:08.362 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.675 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:08.341 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.686 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:08.364 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.824 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:08.504 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.883 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:08.550 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.897 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:08.573 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.978 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:08.656 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.984 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:08.672 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:33.996 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:08.662 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.017 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:08.700 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.038 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:08.726 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.194 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:08.861 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.207 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:08.887 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.214 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:08.890 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.265 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:08.931 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.270 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:08.948 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.325 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:09.013 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.349 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.037 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.376 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.060 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.459 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:09.125 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.480 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.156 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.534 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:09.200 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.539 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:09.217 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.559 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:09.239 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.613 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:09.301 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.673 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.356 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.712 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.400 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.723 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:09.390 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.752 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.428 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.803 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:09.469 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.836 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:09.514 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.850 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:09.530 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.903 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:09.591 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.961 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.644 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.966 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:34.984 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.671 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:34.990 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:09.656 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.023 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.699 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.056+0000 7f61696d4640 10 monclient: get_monmap_and_config 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.056+0000 7f61696d4640 10 monclient: build_initial_monmap 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.056+0000 7f61696d4640 10 monclient: monmap: 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:05:35.056756+0000 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:05:35.056756+0000 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:35.056 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 1 Processor -- start 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 1 -- start start 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient: init 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-a con 0x7f61640a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-b con 0x7f616413f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-c con 0x7f61641430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6164142520 con 0x7f61640a98f0 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61640b28a0 con 0x7f61641430c0 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61640b2a30 con 0x7f616413f710 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.057+0000 7f61696d4640 10 monclient(hunting): authenticate will time out at 6979.355660s 2024-03-25T18:05:35.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.058+0000 7f6162ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f61640a98f0 0x7f61640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52482/0 (socket says 172.21.15.84:52482) 2024-03-25T18:05:35.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.058+0000 7f6162ffd640 1 -- 172.21.15.84:0/2417954818 learned_addr learned my addr 172.21.15.84:0/2417954818 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226694683 0 0) 0x7f6164142520 con 0x7f61640a98f0 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:05:35.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6150003520 con 0x7f61640a98f0 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224619156 0 0) 0x7f61640b2a30 con 0x7f616413f710 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6164142520 con 0x7f616413f710 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088198430 0 0) 0x7f61640b28a0 con 0x7f61641430c0 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:05:35.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.060+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61640b2a30 con 0x7f61641430c0 2024-03-25T18:05:35.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321750786 0 0) 0x7f6150003520 con 0x7f61640a98f0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 >> v1:172.21.15.84:6790/0 conn(0x7f616413f710 legacy=0x7f61641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 >> v1:172.21.15.157:6789/0 conn(0x7f61641430c0 legacy=0x7f616414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 monclient: found mon.noname-a 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640b44c0 con 0x7f61640a98f0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6154001690 con 0x7f61640a98f0 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61627fc640 10 monclient: dump: 2024-03-25T18:05:35.064 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61696d4640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:05:35.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.063+0000 7f61696d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61540017f0 con 0x7f61640a98f0 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 1 -- 172.21.15.84:0/2417954818 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6154001c00 con 0x7f61640a98f0 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 4 set_mon_vals no callback set 2024-03-25T18:05:35.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61627fc640 10 monclient: dump: 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 10 monclient: shutdown 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 1 -- 172.21.15.84:0/2417954818 >> v1:172.21.15.84:6789/0 conn(0x7f61640a98f0 legacy=0x7f61640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 1 -- 172.21.15.84:0/2417954818 shutdown_connections 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 1 -- 172.21.15.84:0/2417954818 >> 172.21.15.84:0/2417954818 conn(0x7f61640b7ea0 msgr2=0x7f61640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 1 -- 172.21.15.84:0/2417954818 shutdown_connections 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.065+0000 7f61696d4640 1 -- 172.21.15.84:0/2417954818 wait complete. 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.066+0000 7f61696d4640 10 monclient: build_initial_monmap 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.066+0000 7f61696d4640 10 monclient: monmap: 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:35.070 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.066+0000 7f61696d4640 1 librados: starting msgr at 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.066+0000 7f61696d4640 1 librados: starting objecter 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.066+0000 7f61696d4640 1 Processor -- start 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 -- start start 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 librados: setting wanted keys 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 librados: calling monclient init 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient: init 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-c con 0x7f61640a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-a con 0x7f616413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient(hunting): picked mon.noname-b con 0x7f61641430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6164132a30 con 0x7f616413f710 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6164138c80 con 0x7f61641430c0 2024-03-25T18:05:35.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6164138f60 con 0x7f61640a98f0 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f61696d4640 10 monclient(hunting): authenticate will time out at 6979.365472s 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f615bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f616413f710 0x7f6164131c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52496/0 (socket says 172.21.15.84:52496) 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.067+0000 7f615bfff640 1 -- 172.21.15.84:0/3678349166 learned_addr learned my addr 172.21.15.84:0/3678349166 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter ms_handle_connect 0x7f616413f710 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter ms_handle_connect 0x7f61640a98f0 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter ms_handle_connect 0x7f61641430c0 2024-03-25T18:05:35.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.068+0000 7f615b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418540960 0 0) 0x7f6164132a30 con 0x7f616413f710 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6144004eb0 con 0x7f616413f710 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909731140 0 0) 0x7f6164138f60 con 0x7f61640a98f0 2024-03-25T18:05:35.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6164132a30 con 0x7f61640a98f0 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743256643 0 0) 0x7f6164138c80 con 0x7f61641430c0 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:05:35.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.070+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6164138f60 con 0x7f61641430c0 2024-03-25T18:05:35.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776571384 0 0) 0x7f6144004eb0 con 0x7f616413f710 2024-03-25T18:05:35.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6164138c80 con 0x7f616413f710 2024-03-25T18:05:35.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f614c001a30 con 0x7f616413f710 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327591992 0 0) 0x7f6164132a30 con 0x7f61640a98f0 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6144004eb0 con 0x7f61640a98f0 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6154001b00 con 0x7f61640a98f0 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.077 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291224620 0 0) 0x7f6164138f60 con 0x7f61641430c0 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.074+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6164132a30 con 0x7f61641430c0 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.075+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f615c001bd0 con 0x7f61641430c0 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.075+0000 7f615b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.075+0000 7f615b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.075+0000 7f615b7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.078 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.079 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.079 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.079 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:09.743 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 377049760 0 0) 0x7f6164138c80 con 0x7f616413f710 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 >> v1:172.21.15.84:6790/0 conn(0x7f61640a98f0 legacy=0x7f6164137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 >> v1:172.21.15.157:6789/0 conn(0x7f61641430c0 legacy=0x7f6164132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 1 monclient: found mon.a 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164139240 con 0x7f616413f710 2024-03-25T18:05:35.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f615b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 10 monclient: _renew_subs 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6164136e50 con 0x7f616413f710 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 10 monclient: _renew_subs 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61641389f0 con 0x7f616413f710 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.077+0000 7f61696d4640 1 librados: init done 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.078+0000 7f61696d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.078+0000 7f61696d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.078+0000 7f61696d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.078+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61640627b0 con 0x7f616413f710 2024-03-25T18:05:35.123 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:09.801 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.138 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:09.818 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.190 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:09.878 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.250 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.933 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.262 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:09.928 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.272 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.960 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.295 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.971 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f614c001c70 con 0x7f616413f710 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f614c003a60 con 0x7f616413f710 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 10 monclient: dump: 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:35.344 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f61617fa640 4 set_mon_vals no callback set 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.344+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f614c003c80 con 0x7f616413f710 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.345+0000 7f6162ffd640 10 monclient: get_auth_request con 0x7f614406e0f0 auth_method 0 2024-03-25T18:05:35.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.345+0000 7f615b7fe640 10 client.4212.objecter ms_handle_connect 0x7f614406e0f0 2024-03-25T18:05:35.347 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:10.013 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f614c0547e0 con 0x7f616413f710 2024-03-25T18:05:35.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 10 client.4212.objecter ms_dispatch 0x7f6164000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f614c0502f0 con 0x7f616413f710 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:35.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.349+0000 7f615b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:35.414 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:10.092 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.452 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.132 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.478 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:10.166 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.523+0000 7f61696d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:35.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.523+0000 7f61696d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:35.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.523+0000 7f61696d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:35.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.523+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61641378f0 con 0x7f616413f710 2024-03-25T18:05:35.526 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:10.193 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.545 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.228 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.562 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.238 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.567 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.255 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.572+0000 7f615b7fe640 1 -- 172.21.15.84:0/3678349166 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f61641378f0 con 0x7f616413f710 2024-03-25T18:05:35.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.572+0000 7f615b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:35.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.572+0000 7f615b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:35.572 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:05:35.572 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:35.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 10 librados: watch_flush enter 2024-03-25T18:05:35.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 10 librados: watch_flush exit 2024-03-25T18:05:35.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:05:35.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 >> v1:172.21.15.157:6800/1801259026 conn(0x7f614406e0f0 legacy=0x7f61440705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 10 monclient: shutdown 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 >> v1:172.21.15.84:6789/0 conn(0x7f616413f710 legacy=0x7f6164131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 shutdown_connections 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 >> 172.21.15.84:0/3678349166 conn(0x7f61640b7ea0 msgr2=0x7f6164074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 shutdown_connections 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 -- 172.21.15.84:0/3678349166 wait complete. 2024-03-25T18:05:35.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:35.574+0000 7f61696d4640 1 librados: shutdown 2024-03-25T18:05:35.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:35.611 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:10.277 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.739 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.420 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.775 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:10.463 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.796 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:10.462 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.827 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.503 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.837 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.520 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.853 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.540 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.943 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:10.620 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:35.997 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:10.674 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.027 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.707 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.093 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:10.759 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.127 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.811 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.150 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.838 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.161 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:10.827 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.302 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.978 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.303 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:10.990 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.313 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:10.979 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.328 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.008 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.356 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:11.032 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.358 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:11.045 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.371 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:11.037 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.454 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.141 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.668 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:11.352 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.722 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:11.406 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.878 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.558 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.947 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.627 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:36.981 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.669 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:37.051 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.739 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:41.540 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:16.227 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:41.582 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:16.249 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:41.616 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:16.293 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:41.618 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:16.284 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:42.199 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:16.875 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:42.390 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:42.405 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:17.088 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:42.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.480+0000 7f982bb5d640 10 monclient: get_monmap_and_config 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.480+0000 7f982bb5d640 10 monclient: build_initial_monmap 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.480+0000 7f982bb5d640 10 monclient: monmap: 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:05:42.481666+0000 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:05:42.481666+0000 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.480+0000 7f982bb5d640 1 Processor -- start 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 1 -- start start 2024-03-25T18:05:42.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient: init 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f98240b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f98240b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f982414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240af940 con 0x7f982414be90 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240b0a70 con 0x7f98240b4d60 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240b0d50 con 0x7f98240b1730 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.481+0000 7f982bb5d640 10 monclient(hunting): authenticate will time out at 6986.780870s 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.482+0000 7f98290f8640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98240b1730 0x7f98240b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:32860/0 (socket says 172.21.15.84:32860) 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.482+0000 7f98290f8640 1 -- 172.21.15.84:0/3210960831 learned_addr learned my addr 172.21.15.84:0/3210960831 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885665434 0 0) 0x7f98240af940 con 0x7f982414be90 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:05:42.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.486+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9810003520 con 0x7f982414be90 2024-03-25T18:05:42.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329482326 0 0) 0x7f98240b0a70 con 0x7f98240b4d60 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98240af940 con 0x7f98240b4d60 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354056323 0 0) 0x7f98240b0d50 con 0x7f98240b1730 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:05:42.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.487+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98240b0a70 con 0x7f98240b1730 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458963304 0 0) 0x7f9810003520 con 0x7f982414be90 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 >> v1:172.21.15.84:6790/0 conn(0x7f98240b1730 legacy=0x7f98240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 >> v1:172.21.15.157:6789/0 conn(0x7f98240b4d60 legacy=0x7f982413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 10 monclient: _finish_hunting 0 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 1 monclient: found mon.noname-a 2024-03-25T18:05:42.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.490+0000 7f98288f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240b1030 con 0x7f982414be90 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: _finish_auth 0 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: _check_auth_tickets 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f981c001a10 con 0x7f982414be90 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f982bb5d640 5 monclient: authenticate success, global_id 4215 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f982bb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f98288f7640 10 monclient: dump: 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.491+0000 7f982bb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.493+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f981c001b70 con 0x7f982414be90 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.493+0000 7f98288f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.493+0000 7f98288f7640 1 -- 172.21.15.84:0/3210960831 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f981c001f80 con 0x7f982414be90 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.493+0000 7f982bb5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.493+0000 7f982bb5d640 4 set_mon_vals no callback set 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f98288f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f98288f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f98288f7640 10 monclient: dump: 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 10 monclient: shutdown 2024-03-25T18:05:42.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 1 -- 172.21.15.84:0/3210960831 >> v1:172.21.15.84:6789/0 conn(0x7f982414be90 legacy=0x7f982414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 1 -- 172.21.15.84:0/3210960831 shutdown_connections 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 1 -- 172.21.15.84:0/3210960831 >> 172.21.15.84:0/3210960831 conn(0x7f98240617a0 msgr2=0x7f9824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 1 -- 172.21.15.84:0/3210960831 shutdown_connections 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.494+0000 7f982bb5d640 1 -- 172.21.15.84:0/3210960831 wait complete. 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.495+0000 7f982bb5d640 10 monclient: build_initial_monmap 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.495+0000 7f982bb5d640 10 monclient: monmap: 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:05:42.502 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 librados: starting msgr at 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 librados: starting objecter 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 Processor -- start 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 -- start start 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 librados: setting wanted keys 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 1 librados: calling monclient init 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.496+0000 7f982bb5d640 10 monclient: init 2024-03-25T18:05:42.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.497+0000 7f982bb5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.497+0000 7f982bb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f98240b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f98240b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f982414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982413bd10 con 0x7f982414be90 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98241dadb0 con 0x7f98240b4d60 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f98298f9640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98240b1730 0x7f9824138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:32868/0 (socket says 172.21.15.84:32868) 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f98298f9640 1 -- 172.21.15.84:0/4143154824 learned_addr learned my addr 172.21.15.84:0/4143154824 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824138950 con 0x7f98240b1730 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.498+0000 7f982bb5d640 10 monclient(hunting): authenticate will time out at 6986.797653s 2024-03-25T18:05:42.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter ms_handle_connect 0x7f982414be90 2024-03-25T18:05:42.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:42.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter ms_handle_connect 0x7f98240b1730 2024-03-25T18:05:42.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:42.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter ms_handle_connect 0x7f98240b4d60 2024-03-25T18:05:42.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.499+0000 7f981a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847109682 0 0) 0x7f982413bd10 con 0x7f982414be90 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.501+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808004eb0 con 0x7f982414be90 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940542944 0 0) 0x7f98241dadb0 con 0x7f98240b4d60 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.502+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982413bd10 con 0x7f98240b4d60 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582887929 0 0) 0x7f9824138950 con 0x7f98240b1730 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.504+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98241dadb0 con 0x7f98240b1730 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197323196 0 0) 0x7f9808004eb0 con 0x7f982414be90 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9824138950 con 0x7f982414be90 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f981c001e80 con 0x7f982414be90 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:42.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.506+0000 7f981a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:42.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1501302161 0 0) 0x7f982413bd10 con 0x7f98240b4d60 2024-03-25T18:05:42.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9808004eb0 con 0x7f98240b4d60 2024-03-25T18:05:42.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f98140017a0 con 0x7f98240b4d60 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:42.514 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3168652818 0 0) 0x7f9824138950 con 0x7f982414be90 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 >> v1:172.21.15.84:6790/0 conn(0x7f98240b1730 legacy=0x7f9824138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 >> v1:172.21.15.157:6789/0 conn(0x7f98240b4d60 legacy=0x7f982413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 monclient: found mon.a 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982413ab60 con 0x7f982414be90 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f981a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 10 monclient: _renew_subs 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98241db9e0 con 0x7f982414be90 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 10 monclient: _renew_subs 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824139890 con 0x7f982414be90 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.509+0000 7f982bb5d640 1 librados: init done 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.510+0000 7f982bb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.510+0000 7f982bb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.510+0000 7f982bb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.510+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982413acf0 con 0x7f982414be90 2024-03-25T18:05:42.582 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:17.269 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f981c002020 con 0x7f982414be90 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f981c003dc0 con 0x7f982414be90 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981a7fc640 10 monclient: dump: 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:42.729 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:42.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.728+0000 7f981bfff640 4 set_mon_vals no callback set 2024-03-25T18:05:42.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.752+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f981c019d90 con 0x7f982414be90 2024-03-25T18:05:42.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.752+0000 7f98298f9640 10 monclient: get_auth_request con 0x7f980806df50 auth_method 0 2024-03-25T18:05:42.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.753+0000 7f981a7fc640 10 client.4218.objecter ms_handle_connect 0x7f980806df50 2024-03-25T18:05:42.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.756+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f981c0537a0 con 0x7f982414be90 2024-03-25T18:05:42.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.756+0000 7f981a7fc640 10 client.4218.objecter ms_dispatch 0x7f9824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:42.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.756+0000 7f981a7fc640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:42.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.756+0000 7f981a7fc640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:42.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.756+0000 7f981a7fc640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T18:05:42.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.847+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f981c04f2b0 con 0x7f982414be90 2024-03-25T18:05:42.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.847+0000 7f981a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:42.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:42.847+0000 7f981a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:43.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.068+0000 7f982bb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:43.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.068+0000 7f982bb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:43.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.068+0000 7f982bb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:05:43.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.068+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98240627b0 con 0x7f982414be90 2024-03-25T18:05:43.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.089+0000 7f981a7fc640 1 -- 172.21.15.84:0/4143154824 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f981c0511e0 con 0x7f982414be90 2024-03-25T18:05:43.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.089+0000 7f981a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:43.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.089+0000 7f981a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:43.090 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:05:43.090 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 10 librados: watch_flush enter 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 10 librados: watch_flush exit 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 10 client.4218.objecter successfully canceled tick 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 >> v1:172.21.15.157:6800/1801259026 conn(0x7f980806df50 legacy=0x7f9808070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 10 monclient: shutdown 2024-03-25T18:05:43.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 >> v1:172.21.15.84:6789/0 conn(0x7f982414be90 legacy=0x7f98241db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 shutdown_connections 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.091+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 >> 172.21.15.84:0/4143154824 conn(0x7f98240617a0 msgr2=0x7f982413c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.092+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 shutdown_connections 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.092+0000 7f982bb5d640 1 -- 172.21.15.84:0/4143154824 wait complete. 2024-03-25T18:05:43.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:43.092+0000 7f982bb5d640 1 librados: shutdown 2024-03-25T18:05:43.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:43.140 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:17.820 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:46.951 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:21.638 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:46.961 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:21.642 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:46.961 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:21.634 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:46.961 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:21.638 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:47.069 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:21.735 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:47.072 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:21.739 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:47.269 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:21.935 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:47.272 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:21.939 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.151 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:22.827 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.155 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:22.831 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.281 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:22.965 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.288 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:22.972 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.375 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:23.063 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.379 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:23.067 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.956 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:23.636 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:48.962 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:23.642 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:49.913 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:50.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.002+0000 7f4d30886640 10 monclient: get_monmap_and_config 2024-03-25T18:05:50.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.002+0000 7f4d30886640 10 monclient: build_initial_monmap 2024-03-25T18:05:50.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.002+0000 7f4d30886640 10 monclient: monmap: 2024-03-25T18:05:50.003 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:50.003 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:05:50.003607+0000 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:05:50.003607+0000 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:50.004 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.003+0000 7f4d30886640 1 Processor -- start 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.003+0000 7f4d30886640 1 -- start start 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.003+0000 7f4d30886640 10 monclient: init 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-a con 0x7f4d2c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:50.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-c con 0x7f4d2c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c142520 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c0b28a0 con 0x7f4d2c1430c0 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c0b2a30 con 0x7f4d2c13f710 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d30886640 10 monclient(hunting): authenticate will time out at 6994.303476s 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d2a59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4d2c0a98f0 0x7f4d2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55284/0 (socket says 172.21.15.84:55284) 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.004+0000 7f4d2a59c640 1 -- 172.21.15.84:0/2050181103 learned_addr learned my addr 172.21.15.84:0/2050181103 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:50.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779995849 0 0) 0x7f4d2c0b2a30 con 0x7f4d2c13f710 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14003520 con 0x7f4d2c13f710 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623730008 0 0) 0x7f4d2c142520 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:05:50.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14003840 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478920992 0 0) 0x7f4d2c0b28a0 con 0x7f4d2c1430c0 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:05:50.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.007+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c142520 con 0x7f4d2c1430c0 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422682470 0 0) 0x7f4d14003840 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 >> v1:172.21.15.84:6790/0 conn(0x7f4d2c13f710 legacy=0x7f4d2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 >> v1:172.21.15.157:6789/0 conn(0x7f4d2c1430c0 legacy=0x7f4d2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 10 monclient: _finish_hunting 0 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 1 monclient: found mon.noname-a 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0b44c0 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 10 monclient: _finish_auth 0 2024-03-25T18:05:50.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 10 monclient: _check_auth_tickets 2024-03-25T18:05:50.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:50.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d30886640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:05:50.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.010+0000 7f4d30886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d1c001630 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: dump: 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.013 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d1c001880 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 1 -- 172.21.15.84:0/2050181103 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d1c001c10 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:50.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d2959a640 10 monclient: dump: 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d30886640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:50.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.012+0000 7f4d30886640 4 set_mon_vals no callback set 2024-03-25T18:05:50.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 10 monclient: shutdown 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 1 -- 172.21.15.84:0/2050181103 >> v1:172.21.15.84:6789/0 conn(0x7f4d2c0a98f0 legacy=0x7f4d2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 1 -- 172.21.15.84:0/2050181103 shutdown_connections 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 1 -- 172.21.15.84:0/2050181103 >> 172.21.15.84:0/2050181103 conn(0x7f4d2c0b7ea0 msgr2=0x7f4d2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 1 -- 172.21.15.84:0/2050181103 shutdown_connections 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.013+0000 7f4d30886640 1 -- 172.21.15.84:0/2050181103 wait complete. 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 10 monclient: build_initial_monmap 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 10 monclient: monmap: 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:05:50.019 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 librados: starting msgr at 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 librados: starting objecter 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 Processor -- start 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 -- start start 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 librados: setting wanted keys 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 1 librados: calling monclient init 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.014+0000 7f4d30886640 10 monclient: init 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:50.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-a con 0x7f4d2c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient(hunting): picked mon.noname-c con 0x7f4d2c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c136f40 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c138c30 con 0x7f4d2c13f710 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c137190 con 0x7f4d2c1430c0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d30886640 10 monclient(hunting): authenticate will time out at 6994.314666s 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d2a59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4d2c0a98f0 0x7f4d2c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55286/0 (socket says 172.21.15.84:55286) 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.015+0000 7f4d2a59c640 1 -- 172.21.15.84:0/2913816778 learned_addr learned my addr 172.21.15.84:0/2913816778 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter ms_handle_connect 0x7f4d2c0a98f0 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter ms_handle_connect 0x7f4d2c13f710 2024-03-25T18:05:50.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:50.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter ms_handle_connect 0x7f4d2c1430c0 2024-03-25T18:05:50.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.016+0000 7f4d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463019069 0 0) 0x7f4d2c136f40 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c004eb0 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066050233 0 0) 0x7f4d2c137190 con 0x7f4d2c1430c0 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c136f40 con 0x7f4d2c1430c0 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184510467 0 0) 0x7f4d2c138c30 con 0x7f4d2c13f710 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:05:50.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.018+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c137190 con 0x7f4d2c13f710 2024-03-25T18:05:50.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 996140518 0 0) 0x7f4d2c137190 con 0x7f4d2c13f710 2024-03-25T18:05:50.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d2c138c30 con 0x7f4d2c13f710 2024-03-25T18:05:50.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790231650 0 0) 0x7f4d2c136f40 con 0x7f4d2c1430c0 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d2c137190 con 0x7f4d2c1430c0 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d24001b90 con 0x7f4d2c1430c0 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): dump: 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.028 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d20001860 con 0x7f4d2c13f710 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.024+0000 7f4d1affd640 10 monclient(hunting): dump: 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.029 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222709936 0 0) 0x7f4d0c004eb0 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d2c136f40 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d1c001b90 con 0x7f4d2c0a98f0 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.025+0000 7f4d1affd640 10 monclient(hunting): dump: 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.031 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2453367444 0 0) 0x7f4d2c138c30 con 0x7f4d2c13f710 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 >> v1:172.21.15.84:6790/0 conn(0x7f4d2c1430c0 legacy=0x7f4d2c0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 >> v1:172.21.15.84:6789/0 conn(0x7f4d2c0a98f0 legacy=0x7f4d2c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 1 monclient: found mon.b 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c139630 con 0x7f4d2c13f710 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 10 monclient: _finish_auth 0 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:50.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 5 monclient: authenticate success, global_id 4225 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 10 monclient: _renew_subs 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c1398e0 con 0x7f4d2c13f710 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 10 monclient: _renew_subs 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c1e3310 con 0x7f4d2c13f710 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.027+0000 7f4d30886640 1 librados: init done 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.028+0000 7f4d30886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.028+0000 7f4d30886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.028+0000 7f4d30886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:50.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.028+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c0627b0 con 0x7f4d2c13f710 2024-03-25T18:05:50.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.304+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d20001cf0 con 0x7f4d2c13f710 2024-03-25T18:05:50.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.304+0000 7f4d1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:50.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.304+0000 7f4d28d99640 4 set_mon_vals no callback set 2024-03-25T18:05:50.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.321+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d20003790 con 0x7f4d2c13f710 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.321+0000 7f4d1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.321+0000 7f4d1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.321+0000 7f4d1affd640 10 monclient: dump: 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:50.322 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.322+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f4d20019760 con 0x7f4d2c13f710 2024-03-25T18:05:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.322+0000 7f4d2a59c640 10 monclient: get_auth_request con 0x7f4d0c06e0a0 auth_method 0 2024-03-25T18:05:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.323+0000 7f4d1affd640 10 client.4225.objecter ms_handle_connect 0x7f4d0c06e0a0 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.331+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f4d200534f0 con 0x7f4d2c13f710 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.331+0000 7f4d1affd640 10 client.4225.objecter ms_dispatch 0x7f4d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.331+0000 7f4d1affd640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.331+0000 7f4d1affd640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.331+0000 7f4d1affd640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.332+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d2004ee30 con 0x7f4d2c13f710 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.332+0000 7f4d1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:50.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.332+0000 7f4d1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:50.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.650+0000 7f4d30886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.650+0000 7f4d30886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.650+0000 7f4d30886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:05:50.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.650+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d2c0ab020 con 0x7f4d2c13f710 2024-03-25T18:05:50.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.670+0000 7f4d1affd640 1 -- 172.21.15.84:0/2913816778 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f4d20050f30 con 0x7f4d2c13f710 2024-03-25T18:05:50.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.670+0000 7f4d1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.670+0000 7f4d1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:50.672 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:05:50.672 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 10 librados: watch_flush enter 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 10 librados: watch_flush exit 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 10 client.4225.objecter successfully canceled tick 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 >> v1:172.21.15.157:6800/1801259026 conn(0x7f4d0c06e0a0 legacy=0x7f4d0c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 10 monclient: shutdown 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 >> v1:172.21.15.157:6789/0 conn(0x7f4d2c13f710 legacy=0x7f4d2c1376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 shutdown_connections 2024-03-25T18:05:50.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.672+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 >> 172.21.15.84:0/2913816778 conn(0x7f4d2c0b7ea0 msgr2=0x7f4d2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:50.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.673+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 shutdown_connections 2024-03-25T18:05:50.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.673+0000 7f4d30886640 1 -- 172.21.15.84:0/2913816778 wait complete. 2024-03-25T18:05:50.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:50.673+0000 7f4d30886640 1 librados: shutdown 2024-03-25T18:05:50.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:51.274 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:00:25.940 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:51.428 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:00:26.116 35445== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:51.580 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:00:26.245 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:51.628 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:00:26.305 35446== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:52.362 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:27.038 35724== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:52.854 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:27.537 35722== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:52.949 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:27.637 35721== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:53.584 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:28.264 35723== Warning: unimplemented fcntl command: 1038 2024-03-25T18:05:57.490 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:57.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.587+0000 7f56d22a1640 10 monclient: get_monmap_and_config 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.587+0000 7f56d22a1640 10 monclient: build_initial_monmap 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.587+0000 7f56d22a1640 10 monclient: monmap: 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:05:57.589323+0000 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:05:57.589323+0000 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:05:57.589 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.589+0000 7f56d22a1640 1 Processor -- start 2024-03-25T18:05:57.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.589+0000 7f56d22a1640 1 -- start start 2024-03-25T18:05:57.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.589+0000 7f56d22a1640 10 monclient: init 2024-03-25T18:05:57.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.589+0000 7f56d22a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:57.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:57.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-a con 0x7f56cc0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:57.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-b con 0x7f56cc0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-c con 0x7f56cc14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0af940 con 0x7f56cc0b4d60 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0b0a70 con 0x7f56cc14be90 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0b0d50 con 0x7f56cc0b1730 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56d22a1640 10 monclient(hunting): authenticate will time out at 7001.889005s 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56cbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f56cc0b4d60 0x7f56cc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55300/0 (socket says 172.21.15.84:55300) 2024-03-25T18:05:57.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.590+0000 7f56cbfff640 1 -- 172.21.15.84:0/1810558571 learned_addr learned my addr 172.21.15.84:0/1810558571 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035044592 0 0) 0x7f56cc0af940 con 0x7f56cc0b4d60 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b4003520 con 0x7f56cc0b4d60 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488074342 0 0) 0x7f56cc0b0a70 con 0x7f56cc14be90 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.594+0000 7f56caffd640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc0af940 con 0x7f56cc14be90 2024-03-25T18:05:57.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546624824 0 0) 0x7f56cc0b0d50 con 0x7f56cc0b1730 2024-03-25T18:05:57.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:05:57.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.595+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc0b0a70 con 0x7f56cc0b1730 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.599+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 110181050 0 0) 0x7f56cc0af940 con 0x7f56cc14be90 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.599+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 >> v1:172.21.15.84:6790/0 conn(0x7f56cc0b1730 legacy=0x7f56cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.599+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 >> v1:172.21.15.84:6789/0 conn(0x7f56cc0b4d60 legacy=0x7f56cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.599+0000 7f56caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.599+0000 7f56caffd640 1 monclient: found mon.noname-c 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:05:57.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0b1030 con 0x7f56cc14be90 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: _finish_auth 0 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56c40019f0 con 0x7f56cc14be90 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56d22a1640 5 monclient: authenticate success, global_id 4228 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56d22a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56caffd640 10 monclient: dump: 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.602 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.603 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.603 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.603 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.603 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.600+0000 7f56d22a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.602+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56c4001c40 con 0x7f56cc14be90 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.602+0000 7f56caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.602+0000 7f56d22a1640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.602+0000 7f56d22a1640 4 set_mon_vals no callback set 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.602+0000 7f56caffd640 1 -- 172.21.15.84:0/1810558571 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56c4001fd0 con 0x7f56cc14be90 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56caffd640 10 monclient: dump: 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.605 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 10 monclient: shutdown 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 1 -- 172.21.15.84:0/1810558571 >> v1:172.21.15.157:6789/0 conn(0x7f56cc14be90 legacy=0x7f56cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 1 -- 172.21.15.84:0/1810558571 shutdown_connections 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 1 -- 172.21.15.84:0/1810558571 >> 172.21.15.84:0/1810558571 conn(0x7f56cc0617a0 msgr2=0x7f56cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 1 -- 172.21.15.84:0/1810558571 shutdown_connections 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.603+0000 7f56d22a1640 1 -- 172.21.15.84:0/1810558571 wait complete. 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.604+0000 7f56d22a1640 10 monclient: build_initial_monmap 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.604+0000 7f56d22a1640 10 monclient: monmap: 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:57.606 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:05:57.607 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.604+0000 7f56d22a1640 1 librados: starting msgr at 2024-03-25T18:05:57.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.604+0000 7f56d22a1640 1 librados: starting objecter 2024-03-25T18:05:57.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.605+0000 7f56d22a1640 1 Processor -- start 2024-03-25T18:05:57.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.605+0000 7f56d22a1640 1 -- start start 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.605+0000 7f56d22a1640 1 librados: setting wanted keys 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.605+0000 7f56d22a1640 1 librados: calling monclient init 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.605+0000 7f56d22a1640 10 monclient: init 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-a con 0x7f56cc0b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-b con 0x7f56cc0b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient(hunting): picked mon.noname-c con 0x7f56cc14be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc143320 con 0x7f56cc0b1730 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56cbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f56cc0b1730 0x7f56cc1dc0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55306/0 (socket says 172.21.15.84:55306) 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56cbfff640 1 -- 172.21.15.84:0/127461987 learned_addr learned my addr 172.21.15.84:0/127461987 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc137fc0 con 0x7f56cc0b4d60 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc06ef00 con 0x7f56cc14be90 2024-03-25T18:05:57.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.606+0000 7f56d22a1640 10 monclient(hunting): authenticate will time out at 7001.905705s 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.607+0000 7f56c8ff9640 10 client.?.objecter ms_handle_connect 0x7f56cc0b1730 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.607+0000 7f56c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.607+0000 7f56c8ff9640 10 client.?.objecter ms_handle_connect 0x7f56cc0b4d60 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.607+0000 7f56c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.608+0000 7f56c8ff9640 10 client.?.objecter ms_handle_connect 0x7f56cc14be90 2024-03-25T18:05:57.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.608+0000 7f56c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056630967 0 0) 0x7f56cc137fc0 con 0x7f56cc0b4d60 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): my global_id is 4231 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b8004eb0 con 0x7f56cc0b4d60 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169282064 0 0) 0x7f56cc143320 con 0x7f56cc0b1730 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 10 monclient(hunting): my global_id is 4230 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.610+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc137fc0 con 0x7f56cc0b1730 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42406530 0 0) 0x7f56cc06ef00 con 0x7f56cc14be90 2024-03-25T18:05:57.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 10 monclient(hunting): my global_id is 4232 2024-03-25T18:05:57.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.611+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc143320 con 0x7f56cc14be90 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2838237570 0 0) 0x7f56b8004eb0 con 0x7f56cc0b4d60 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56cc06ef00 con 0x7f56cc0b4d60 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56c00019b0 con 0x7f56cc0b4d60 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.618 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.617+0000 7f56c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:57.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681685987 0 0) 0x7f56cc137fc0 con 0x7f56cc0b1730 2024-03-25T18:05:57.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56b8004eb0 con 0x7f56cc0b1730 2024-03-25T18:05:57.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56bc001830 con 0x7f56cc0b1730 2024-03-25T18:05:57.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3982157950 0 0) 0x7f56cc143320 con 0x7f56cc14be90 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56cc137fc0 con 0x7f56cc14be90 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56c40019b0 con 0x7f56cc14be90 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.618+0000 7f56c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.622 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 43075907 0 0) 0x7f56cc137fc0 con 0x7f56cc14be90 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 >> v1:172.21.15.157:6789/0 conn(0x7f56cc0b4d60 legacy=0x7f56cc1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 >> v1:172.21.15.84:6789/0 conn(0x7f56cc0b1730 legacy=0x7f56cc1dc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 1 monclient: found mon.c 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc06f1e0 con 0x7f56cc14be90 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.623+0000 7f56c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:57.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 5 monclient: authenticate success, global_id 4232 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 10 monclient: _renew_subs 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc142f20 con 0x7f56cc14be90 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 10 monclient: _renew_subs 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc06e080 con 0x7f56cc14be90 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.624+0000 7f56d22a1640 1 librados: init done 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.625+0000 7f56d22a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.625+0000 7f56d22a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.625+0000 7f56d22a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:57.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.625+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc0b3cd0 con 0x7f56cc14be90 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56c4001400 con 0x7f56cc14be90 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f56c4003dc0 con 0x7f56cc14be90 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56c8ff9640 10 monclient: dump: 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:05:57.947 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:05:57.948 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:05:57.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.946+0000 7f56ca7fc640 4 set_mon_vals no callback set 2024-03-25T18:05:57.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.947+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f56c4019d90 con 0x7f56cc14be90 2024-03-25T18:05:57.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.948+0000 7f56cbfff640 10 monclient: get_auth_request con 0x7f56b806e060 auth_method 0 2024-03-25T18:05:57.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.948+0000 7f56c8ff9640 10 client.4232.objecter ms_handle_connect 0x7f56b806e060 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f56c4053870 con 0x7f56cc14be90 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 10 client.4232.objecter ms_dispatch 0x7f56cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56c404f1b0 con 0x7f56cc14be90 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:57.973+0000 7f56c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:58.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.350+0000 7f56d22a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:58.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.350+0000 7f56d22a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:58.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.350+0000 7f56d22a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:05:58.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.350+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56cc0627b0 con 0x7f56cc14be90 2024-03-25T18:05:58.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.373+0000 7f56c8ff9640 1 -- 172.21.15.84:0/127461987 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f56c40512b0 con 0x7f56cc14be90 2024-03-25T18:05:58.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.373+0000 7f56c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:58.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.373+0000 7f56c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:58.375 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:05:58.375 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 10 librados: watch_flush enter 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 10 librados: watch_flush exit 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 10 client.4232.objecter successfully canceled tick 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 >> v1:172.21.15.157:6800/1801259026 conn(0x7f56b806e060 legacy=0x7f56b8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 10 monclient: shutdown 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 >> v1:172.21.15.84:6790/0 conn(0x7f56cc14be90 legacy=0x7f56cc1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 shutdown_connections 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.376+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 >> 172.21.15.84:0/127461987 conn(0x7f56cc0617a0 msgr2=0x7f56cc0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:58.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.377+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 shutdown_connections 2024-03-25T18:05:58.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.377+0000 7f56d22a1640 1 -- 172.21.15.84:0/127461987 wait complete. 2024-03-25T18:05:58.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:05:58.377+0000 7f56d22a1640 1 librados: shutdown 2024-03-25T18:05:58.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:59.843 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-25T18:05:59.839+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:06:00.230 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-25T18:06:00.133+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:06:00.346 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-25T18:06:00.342+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:06:00.524 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-25T18:06:00.520+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:06:00.969 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-25T18:06:00.964+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:06:01.497 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-25T18:06:01.492+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:06:02.053 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-25T18:06:02.048+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:06:02.502 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-25T18:06:02.497+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:06:05.199 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.315+0000 7f0b2edc6640 10 monclient: get_monmap_and_config 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.315+0000 7f0b2edc6640 10 monclient: build_initial_monmap 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.315+0000 7f0b2edc6640 10 monclient: monmap: 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:05.316540+0000 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:05.316540+0000 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.316 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:05.317 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:05.317 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:05.317 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 1 Processor -- start 2024-03-25T18:06:05.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 1 -- start start 2024-03-25T18:06:05.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient: init 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b280a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b281430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28142520 con 0x7f0b281430c0 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280b28a0 con 0x7f0b280a98f0 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280b2a30 con 0x7f0b2813f710 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:05.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.316+0000 7f0b2edc6640 10 monclient(hunting): authenticate will time out at 7009.615613s 2024-03-25T18:06:05.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.317+0000 7f0b2cb62640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0b280a98f0 0x7f0b280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59992/0 (socket says 172.21.15.84:59992) 2024-03-25T18:06:05.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.317+0000 7f0b2cb62640 1 -- 172.21.15.84:0/3198402531 learned_addr learned my addr 172.21.15.84:0/3198402531 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.386+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813181446 0 0) 0x7f0b280b28a0 con 0x7f0b280a98f0 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.387+0000 7f0b277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.387+0000 7f0b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.387+0000 7f0b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.387+0000 7f0b277fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:06:05.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.387+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18003520 con 0x7f0b280a98f0 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442381903 0 0) 0x7f0b280b2a30 con 0x7f0b2813f710 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:06:05.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.388+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b280b28a0 con 0x7f0b2813f710 2024-03-25T18:06:05.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635849209 0 0) 0x7f0b28142520 con 0x7f0b281430c0 2024-03-25T18:06:05.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:06:05.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.393+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b280b2a30 con 0x7f0b281430c0 2024-03-25T18:06:05.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3739534987 0 0) 0x7f0b18003520 con 0x7f0b280a98f0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 >> v1:172.21.15.84:6790/0 conn(0x7f0b2813f710 legacy=0x7f0b281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 >> v1:172.21.15.84:6789/0 conn(0x7f0b281430c0 legacy=0x7f0b2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 monclient: found mon.noname-c 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280b44c0 con 0x7f0b280a98f0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0b1c001690 con 0x7f0b280a98f0 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b2edc6640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b2edc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b277fe640 10 monclient: dump: 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.400 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.401 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:05.401 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:05.401 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:05.401 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.397+0000 7f0b2edc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:05.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c0017f0 con 0x7f0b280a98f0 2024-03-25T18:06:05.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 1 -- 172.21.15.84:0/3198402531 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0b1c001c00 con 0x7f0b280a98f0 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b2edc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b2edc6640 4 set_mon_vals no callback set 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.409+0000 7f0b277fe640 10 monclient: dump: 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.411 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.412 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.412 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:05.412 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:05.412 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:05.412 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 10 monclient: shutdown 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 1 -- 172.21.15.84:0/3198402531 >> v1:172.21.15.157:6789/0 conn(0x7f0b280a98f0 legacy=0x7f0b280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 1 -- 172.21.15.84:0/3198402531 shutdown_connections 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 1 -- 172.21.15.84:0/3198402531 >> 172.21.15.84:0/3198402531 conn(0x7f0b280b7ea0 msgr2=0x7f0b280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 1 -- 172.21.15.84:0/3198402531 shutdown_connections 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.410+0000 7f0b2edc6640 1 -- 172.21.15.84:0/3198402531 wait complete. 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.411+0000 7f0b2edc6640 10 monclient: build_initial_monmap 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.411+0000 7f0b2edc6640 10 monclient: monmap: 2024-03-25T18:06:05.415 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.411+0000 7f0b2edc6640 1 librados: starting msgr at 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.411+0000 7f0b2edc6640 1 librados: starting objecter 2024-03-25T18:06:05.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.412+0000 7f0b2edc6640 1 Processor -- start 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.412+0000 7f0b2edc6640 1 -- start start 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.412+0000 7f0b2edc6640 1 librados: setting wanted keys 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.412+0000 7f0b2edc6640 1 librados: calling monclient init 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.412+0000 7f0b2edc6640 10 monclient: init 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b280a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b281430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28138250 con 0x7f0b280a98f0 2024-03-25T18:06:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28137d50 con 0x7f0b281430c0 2024-03-25T18:06:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28137ee0 con 0x7f0b2813f710 2024-03-25T18:06:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.413+0000 7f0b2edc6640 10 monclient(hunting): authenticate will time out at 7009.712527s 2024-03-25T18:06:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.416+0000 7f0b2d363640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0b281430c0 0x7f0b281d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:60042/0 (socket says 172.21.15.84:60042) 2024-03-25T18:06:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.416+0000 7f0b2d363640 1 -- 172.21.15.84:0/1886351538 learned_addr learned my addr 172.21.15.84:0/1886351538 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:05.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.417+0000 7f0b257fa640 10 client.?.objecter ms_handle_connect 0x7f0b281430c0 2024-03-25T18:06:05.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.417+0000 7f0b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:05.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.419+0000 7f0b257fa640 10 client.?.objecter ms_handle_connect 0x7f0b2813f710 2024-03-25T18:06:05.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.419+0000 7f0b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761891390 0 0) 0x7f0b28137d50 con 0x7f0b281430c0 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:06:05.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.426+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c004eb0 con 0x7f0b281430c0 2024-03-25T18:06:05.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.427+0000 7f0b257fa640 10 client.?.objecter ms_handle_connect 0x7f0b280a98f0 2024-03-25T18:06:05.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.427+0000 7f0b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:05.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959139228 0 0) 0x7f0b28137ee0 con 0x7f0b2813f710 2024-03-25T18:06:05.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:06:05.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.430+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b28137d50 con 0x7f0b2813f710 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.439+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 201045688 0 0) 0x7f0b28137d50 con 0x7f0b2813f710 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.439+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b28137ee0 con 0x7f0b2813f710 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.439+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0b14001970 con 0x7f0b2813f710 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.439+0000 7f0b257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.440+0000 7f0b257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.440+0000 7f0b257fa640 10 monclient(hunting): dump: 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.441 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.440+0000 7f0b257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:05.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075431819 0 0) 0x7f0b0c004eb0 con 0x7f0b281430c0 2024-03-25T18:06:05.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b28137d50 con 0x7f0b281430c0 2024-03-25T18:06:05.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0b20001c70 con 0x7f0b281430c0 2024-03-25T18:06:05.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.447+0000 7f0b257fa640 10 monclient(hunting): dump: 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:05.449 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:05.450 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298366041 0 0) 0x7f0b28138250 con 0x7f0b280a98f0 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:06:05.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.448+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c004eb0 con 0x7f0b280a98f0 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1175785757 0 0) 0x7f0b28137ee0 con 0x7f0b2813f710 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 >> v1:172.21.15.157:6789/0 conn(0x7f0b281430c0 legacy=0x7f0b281d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 >> v1:172.21.15.84:6789/0 conn(0x7f0b280a98f0 legacy=0x7f0b28137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 1 monclient: found mon.c 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280b7730 con 0x7f0b2813f710 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 10 monclient: _finish_auth 0 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:06:05.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b2edc6640 5 monclient: authenticate success, global_id 4262 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.449+0000 7f0b2edc6640 10 monclient: _renew_subs 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b280b7920 con 0x7f0b2813f710 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 monclient: _renew_subs 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b281e33d0 con 0x7f0b2813f710 2024-03-25T18:06:05.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 1 librados: init done 2024-03-25T18:06:05.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:05.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:05.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:05.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.450+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b280627b0 con 0x7f0b2813f710 2024-03-25T18:06:05.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b14003610 con 0x7f0b2813f710 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0b14003920 con 0x7f0b2813f710 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b257fa640 10 monclient: dump: 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:05.756 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:05.757 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:05.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.754+0000 7f0b26ffd640 4 set_mon_vals no callback set 2024-03-25T18:06:05.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.755+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f0b140198f0 con 0x7f0b2813f710 2024-03-25T18:06:05.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.756+0000 7f0b2cb62640 10 monclient: get_auth_request con 0x7f0b0c06df00 auth_method 0 2024-03-25T18:06:05.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.756+0000 7f0b257fa640 10 client.4262.objecter ms_handle_connect 0x7f0b0c06df00 2024-03-25T18:06:05.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.761+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f0b140535e0 con 0x7f0b2813f710 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.761+0000 7f0b257fa640 10 client.4262.objecter ms_dispatch 0x7f0b28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.761+0000 7f0b257fa640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.761+0000 7f0b257fa640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.762+0000 7f0b257fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.762+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1404ef20 con 0x7f0b2813f710 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.762+0000 7f0b257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:05.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:05.762+0000 7f0b257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:06.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.013+0000 7f0b2edc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:06.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.013+0000 7f0b2edc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:06.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.013+0000 7f0b2edc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:06.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.013+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b28138400 con 0x7f0b2813f710 2024-03-25T18:06:06.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.031+0000 7f0b257fa640 1 -- 172.21.15.84:0/1886351538 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f0b28138400 con 0x7f0b2813f710 2024-03-25T18:06:06.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.031+0000 7f0b257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:06.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.031+0000 7f0b257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:06.032 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:06.032 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 10 librados: watch_flush enter 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 10 librados: watch_flush exit 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 10 client.4262.objecter successfully canceled tick 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 >> v1:172.21.15.157:6800/1801259026 conn(0x7f0b0c06df00 legacy=0x7f0b0c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 10 monclient: shutdown 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 >> v1:172.21.15.84:6790/0 conn(0x7f0b2813f710 legacy=0x7f0b281d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 shutdown_connections 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 >> 172.21.15.84:0/1886351538 conn(0x7f0b280b7ea0 msgr2=0x7f0b28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 shutdown_connections 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.033+0000 7f0b2edc6640 1 -- 172.21.15.84:0/1886351538 wait complete. 2024-03-25T18:06:06.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:06.034+0000 7f0b2edc6640 1 librados: shutdown 2024-03-25T18:06:06.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:12.853 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.969+0000 7f13e5b3d640 10 monclient: get_monmap_and_config 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.969+0000 7f13e5b3d640 10 monclient: build_initial_monmap 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.969+0000 7f13e5b3d640 10 monclient: monmap: 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:12.969 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:12.970162+0000 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:12.970162+0000 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.970+0000 7f13e5b3d640 1 Processor -- start 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.970+0000 7f13e5b3d640 1 -- start start 2024-03-25T18:06:12.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.970+0000 7f13e5b3d640 10 monclient: init 2024-03-25T18:06:12.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:12.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:12.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f13e00a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:12.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f13e013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f13e01430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e0142520 con 0x7f13e013f710 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b28a0 con 0x7f13e00a98f0 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b2a30 con 0x7f13e01430c0 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:12.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.971+0000 7f13e5b3d640 10 monclient(hunting): authenticate will time out at 7017.269687s 2024-03-25T18:06:12.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.978+0000 7f13df7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f13e00a98f0 0x7f13e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56062/0 (socket says 172.21.15.84:56062) 2024-03-25T18:06:12.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.978+0000 7f13df7fe640 1 -- 172.21.15.84:0/3470413823 learned_addr learned my addr 172.21.15.84:0/3470413823 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:12.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846546843 0 0) 0x7f13e00b28a0 con 0x7f13e00a98f0 2024-03-25T18:06:12.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:12.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:12.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:12.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:06:12.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:12.986+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13e00a98f0 2024-03-25T18:06:13.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896772196 0 0) 0x7f13e0142520 con 0x7f13e013f710 2024-03-25T18:06:13.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:13.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:13.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:13.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:06:13.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.011+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e00b28a0 con 0x7f13e013f710 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500315263 0 0) 0x7f13e00b2a30 con 0x7f13e01430c0 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:06:13.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.029+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e0142520 con 0x7f13e01430c0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4028623832 0 0) 0x7f13e0142520 con 0x7f13e01430c0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 >> v1:172.21.15.157:6789/0 conn(0x7f13e00a98f0 legacy=0x7f13e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 >> v1:172.21.15.84:6789/0 conn(0x7f13e013f710 legacy=0x7f13e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 1 monclient: found mon.noname-b 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.095+0000 7f13de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00b44c0 con 0x7f13e01430c0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:13.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d8001930 con 0x7f13e01430c0 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13de7fc640 10 monclient: dump: 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:13.097 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13e5b3d640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:06:13.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.096+0000 7f13e5b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d8001a90 con 0x7f13e01430c0 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 1 -- 172.21.15.84:0/3470413823 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d8001ee0 con 0x7f13e01430c0 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13e5b3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13e5b3d640 4 set_mon_vals no callback set 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13de7fc640 10 monclient: dump: 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.115 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:13.116 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.115+0000 7f13e5b3d640 10 monclient: shutdown 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 1 -- 172.21.15.84:0/3470413823 >> v1:172.21.15.84:6790/0 conn(0x7f13e01430c0 legacy=0x7f13e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 1 -- 172.21.15.84:0/3470413823 shutdown_connections 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 1 -- 172.21.15.84:0/3470413823 >> 172.21.15.84:0/3470413823 conn(0x7f13e00b7ea0 msgr2=0x7f13e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:13.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 1 -- 172.21.15.84:0/3470413823 shutdown_connections 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 1 -- 172.21.15.84:0/3470413823 wait complete. 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 10 monclient: build_initial_monmap 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.116+0000 7f13e5b3d640 10 monclient: monmap: 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:13.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 librados: starting msgr at 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 librados: starting objecter 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 Processor -- start 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 -- start start 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 librados: setting wanted keys 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 1 librados: calling monclient init 2024-03-25T18:06:13.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.117+0000 7f13e5b3d640 10 monclient: init 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f13e00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f13e013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f13e01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01dae50 con 0x7f13e00a98f0 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e0138ee0 con 0x7f13e01430c0 2024-03-25T18:06:13.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01daa20 con 0x7f13e013f710 2024-03-25T18:06:13.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:13.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.118+0000 7f13e5b3d640 10 monclient(hunting): authenticate will time out at 7017.416139s 2024-03-25T18:06:13.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.125+0000 7f13dffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f13e01430c0 0x7f13e01e2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56114/0 (socket says 172.21.15.84:56114) 2024-03-25T18:06:13.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.125+0000 7f13dffff640 1 -- 172.21.15.84:0/1168085465 learned_addr learned my addr 172.21.15.84:0/1168085465 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:13.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.127+0000 7f13c7fff640 10 client.?.objecter ms_handle_connect 0x7f13e01430c0 2024-03-25T18:06:13.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.127+0000 7f13c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:13.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.131+0000 7f13c7fff640 10 client.?.objecter ms_handle_connect 0x7f13e013f710 2024-03-25T18:06:13.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.131+0000 7f13c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:13.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.131+0000 7f13c7fff640 10 client.?.objecter ms_handle_connect 0x7f13e00a98f0 2024-03-25T18:06:13.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.131+0000 7f13c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:13.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025659925 0 0) 0x7f13e0138ee0 con 0x7f13e01430c0 2024-03-25T18:06:13.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:13.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:13.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:13.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:06:13.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.135+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13cc004eb0 con 0x7f13e01430c0 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326653664 0 0) 0x7f13e01daa20 con 0x7f13e013f710 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:06:13.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.147+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e0138ee0 con 0x7f13e013f710 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198841564 0 0) 0x7f13e01dae50 con 0x7f13e00a98f0 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:06:13.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.148+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e01daa20 con 0x7f13e00a98f0 2024-03-25T18:06:13.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964852598 0 0) 0x7f13cc004eb0 con 0x7f13e01430c0 2024-03-25T18:06:13.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e01dae50 con 0x7f13e01430c0 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d8001ea0 con 0x7f13e01430c0 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:13.150 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.149+0000 7f13c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:13.156 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.156+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659038077 0 0) 0x7f13e0138ee0 con 0x7f13e013f710 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.156+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13cc004eb0 con 0x7f13e013f710 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.157+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d4001930 con 0x7f13e013f710 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.157+0000 7f13c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.157+0000 7f13c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.157+0000 7f13c7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:13.157 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116194118 0 0) 0x7f13e01dae50 con 0x7f13e01430c0 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 >> v1:172.21.15.84:6790/0 conn(0x7f13e013f710 legacy=0x7f13e01387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 >> v1:172.21.15.84:6789/0 conn(0x7f13e00a98f0 legacy=0x7f13e01db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 1 monclient: found mon.b 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:13.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e0137f80 con 0x7f13e01430c0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 10 monclient: _renew_subs 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e01381d0 con 0x7f13e01430c0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 10 monclient: _renew_subs 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e01352f0 con 0x7f13e01430c0 2024-03-25T18:06:13.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.168+0000 7f13e5b3d640 1 librados: init done 2024-03-25T18:06:13.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.169+0000 7f13e5b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:13.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.169+0000 7f13e5b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:13.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.169+0000 7f13e5b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:13.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.169+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e00b3ef0 con 0x7f13e01430c0 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d8003c20 con 0x7f13e01430c0 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d8002100 con 0x7f13e01430c0 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13c7fff640 10 monclient: dump: 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:13.469 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.468+0000 7f13ddffb640 4 set_mon_vals no callback set 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.469+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f13d8004070 con 0x7f13e01430c0 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.470+0000 7f13df7fe640 10 monclient: get_auth_request con 0x7f13cc06df70 auth_method 0 2024-03-25T18:06:13.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.470+0000 7f13c7fff640 10 client.4267.objecter ms_handle_connect 0x7f13cc06df70 2024-03-25T18:06:13.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.475+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f13d8054c70 con 0x7f13e01430c0 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.475+0000 7f13c7fff640 10 client.4267.objecter ms_dispatch 0x7f13e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.475+0000 7f13c7fff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.475+0000 7f13c7fff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.475+0000 7f13c7fff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.476+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13d8050780 con 0x7f13e01430c0 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.476+0000 7f13c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:13.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.476+0000 7f13c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:13.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.724+0000 7f13e5b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.724+0000 7f13e5b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.724+0000 7f13e5b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:13.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.724+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13e01db000 con 0x7f13e01430c0 2024-03-25T18:06:13.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.761+0000 7f13c7fff640 1 -- 172.21.15.84:0/1168085465 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f13e01db000 con 0x7f13e01430c0 2024-03-25T18:06:13.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.761+0000 7f13c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.761+0000 7f13c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:13.762 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:13.762 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:13.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 10 librados: watch_flush enter 2024-03-25T18:06:13.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 10 librados: watch_flush exit 2024-03-25T18:06:13.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T18:06:13.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 10 client.4267.objecter successfully canceled tick 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 >> v1:172.21.15.157:6800/1801259026 conn(0x7f13cc06df70 legacy=0x7f13cc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 10 monclient: shutdown 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 >> v1:172.21.15.157:6789/0 conn(0x7f13e01430c0 legacy=0x7f13e01e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 shutdown_connections 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 >> 172.21.15.84:0/1168085465 conn(0x7f13e00b7ea0 msgr2=0x7f13e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 shutdown_connections 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 -- 172.21.15.84:0/1168085465 wait complete. 2024-03-25T18:06:13.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:13.764+0000 7f13e5b3d640 1 librados: shutdown 2024-03-25T18:06:13.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:17.474 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:14.293 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.550 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:14.369 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.573 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:14.357 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.596 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:14.415 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.629 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:14.413 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.660 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:14.480 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.797 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:14.617 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.848 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:14.632 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:17.898 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:14.718 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:20.581 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:20.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.676+0000 7fe30f16d640 10 monclient: get_monmap_and_config 2024-03-25T18:06:20.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.676+0000 7fe30f16d640 10 monclient: build_initial_monmap 2024-03-25T18:06:20.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.676+0000 7fe30f16d640 10 monclient: monmap: 2024-03-25T18:06:20.676 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:20.676 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:20.677097+0000 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:20.677097+0000 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:20.677 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.677+0000 7fe30f16d640 1 Processor -- start 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.677+0000 7fe30f16d640 1 -- start start 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.677+0000 7fe30f16d640 10 monclient: init 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3080a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:20.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-c con 0x7fe30813f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3081430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe308142520 con 0x7fe3081430c0 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080b28a0 con 0x7fe30813f710 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080b2a30 con 0x7fe3080a98f0 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:20.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.678+0000 7fe30f16d640 10 monclient(hunting): authenticate will time out at 7024.976062s 2024-03-25T18:06:20.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.682+0000 7fe30cf09640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe3080a98f0 0x7fe3080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41744/0 (socket says 172.21.15.84:41744) 2024-03-25T18:06:20.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.682+0000 7fe30cf09640 1 -- 172.21.15.84:0/4231883933 learned_addr learned my addr 172.21.15.84:0/4231883933 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047452041 0 0) 0x7fe3080b2a30 con 0x7fe3080a98f0 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:06:20.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.703+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8003520 con 0x7fe3080a98f0 2024-03-25T18:06:20.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262308849 0 0) 0x7fe3080b28a0 con 0x7fe30813f710 2024-03-25T18:06:20.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:06:20.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.719+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3080b2a30 con 0x7fe30813f710 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19005835 0 0) 0x7fe2f8003520 con 0x7fe3080a98f0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 >> v1:172.21.15.157:6789/0 conn(0x7fe30813f710 legacy=0x7fe3081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 >> v1:172.21.15.84:6789/0 conn(0x7fe3081430c0 legacy=0x7fe30814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 monclient: found mon.noname-b 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080b44c0 con 0x7fe3080a98f0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2fc001690 con 0x7fe3080a98f0 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:20.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe3077fe640 10 monclient: dump: 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe30f16d640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:06:20.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.723+0000 7fe30f16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2fc0017f0 con 0x7fe3080a98f0 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 1 -- 172.21.15.84:0/4231883933 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2fc001c00 con 0x7fe3080a98f0 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe3077fe640 10 monclient: dump: 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.754 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe30f16d640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:20.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.754+0000 7fe30f16d640 4 set_mon_vals no callback set 2024-03-25T18:06:20.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 10 monclient: shutdown 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 1 -- 172.21.15.84:0/4231883933 >> v1:172.21.15.84:6790/0 conn(0x7fe3080a98f0 legacy=0x7fe3080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 1 -- 172.21.15.84:0/4231883933 shutdown_connections 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 1 -- 172.21.15.84:0/4231883933 >> 172.21.15.84:0/4231883933 conn(0x7fe3080b7ea0 msgr2=0x7fe3080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 1 -- 172.21.15.84:0/4231883933 shutdown_connections 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.755+0000 7fe30f16d640 1 -- 172.21.15.84:0/4231883933 wait complete. 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.756+0000 7fe30f16d640 10 monclient: build_initial_monmap 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.756+0000 7fe30f16d640 10 monclient: monmap: 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:20.757 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.756+0000 7fe30f16d640 1 librados: starting msgr at 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.756+0000 7fe30f16d640 1 librados: starting objecter 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.756+0000 7fe30f16d640 1 Processor -- start 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.757+0000 7fe30f16d640 1 -- start start 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.757+0000 7fe30f16d640 1 librados: setting wanted keys 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.757+0000 7fe30f16d640 1 librados: calling monclient init 2024-03-25T18:06:20.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.757+0000 7fe30f16d640 10 monclient: init 2024-03-25T18:06:20.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:20.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:20.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3080a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-c con 0x7fe30813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3081430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe308138b20 con 0x7fe3081430c0 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe308137a80 con 0x7fe3080a98f0 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe308139880 con 0x7fe30813f710 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:20.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.758+0000 7fe30f16d640 10 monclient(hunting): authenticate will time out at 7025.056192s 2024-03-25T18:06:20.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.762+0000 7fe30cf09640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe3080a98f0 0x7fe308137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35714/0 (socket says 172.21.15.84:35714) 2024-03-25T18:06:20.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.762+0000 7fe30cf09640 1 -- 172.21.15.84:0/756707912 learned_addr learned my addr 172.21.15.84:0/756707912 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:20.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.764+0000 7fe3057fa640 10 client.?.objecter ms_handle_connect 0x7fe3081430c0 2024-03-25T18:06:20.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.764+0000 7fe3057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.764+0000 7fe3057fa640 10 client.?.objecter ms_handle_connect 0x7fe3080a98f0 2024-03-25T18:06:20.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.764+0000 7fe3057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373567789 0 0) 0x7fe308137a80 con 0x7fe3080a98f0 2024-03-25T18:06:20.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:06:20.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.772+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec004eb0 con 0x7fe3080a98f0 2024-03-25T18:06:20.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.774+0000 7fe3057fa640 10 client.?.objecter ms_handle_connect 0x7fe30813f710 2024-03-25T18:06:20.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.774+0000 7fe3057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019903848 0 0) 0x7fe308138b20 con 0x7fe3081430c0 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:06:20.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.779+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308137a80 con 0x7fe3081430c0 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748125512 0 0) 0x7fe308139880 con 0x7fe30813f710 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:06:20.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.784+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308138b20 con 0x7fe30813f710 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865904959 0 0) 0x7fe2ec004eb0 con 0x7fe3080a98f0 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe308139880 con 0x7fe3080a98f0 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2fc001bc0 con 0x7fe3080a98f0 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:20.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 10 monclient(hunting): dump: 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.796+0000 7fe3057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:20.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 205172194 0 0) 0x7fe308138b20 con 0x7fe30813f710 2024-03-25T18:06:20.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2ec004eb0 con 0x7fe30813f710 2024-03-25T18:06:20.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2f4001930 con 0x7fe30813f710 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): dump: 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348388991 0 0) 0x7fe308137a80 con 0x7fe3081430c0 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe308138b20 con 0x7fe3081430c0 2024-03-25T18:06:20.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe300001b10 con 0x7fe3081430c0 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.801+0000 7fe3057fa640 10 monclient(hunting): dump: 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3007839729 0 0) 0x7fe308139880 con 0x7fe3080a98f0 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 >> v1:172.21.15.84:6790/0 conn(0x7fe30813f710 legacy=0x7fe308139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 >> v1:172.21.15.84:6789/0 conn(0x7fe3081430c0 legacy=0x7fe3081e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 10 monclient: _finish_hunting 0 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 1 monclient: found mon.b 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308139a10 con 0x7fe3080a98f0 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 10 monclient: _finish_auth 0 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 10 monclient: _check_auth_tickets 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe3057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:20.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 10 monclient: _renew_subs 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3081dab00 con 0x7fe3080a98f0 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 10 monclient: _renew_subs 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30813b4f0 con 0x7fe3080a98f0 2024-03-25T18:06:20.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.816+0000 7fe30f16d640 1 librados: init done 2024-03-25T18:06:20.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.817+0000 7fe30f16d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:20.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.818+0000 7fe30f16d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:20.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.818+0000 7fe30f16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:20.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.818+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe308074cc0 con 0x7fe3080a98f0 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2fc0011f0 con 0x7fe3080a98f0 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe2fc004cb0 con 0x7fe3080a98f0 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe3057fa640 10 monclient: dump: 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.844 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.844+0000 7fe306ffd640 4 set_mon_vals no callback set 2024-03-25T18:06:20.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.845+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fe2fc01ad00 con 0x7fe3080a98f0 2024-03-25T18:06:20.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.845+0000 7fe307fff640 10 monclient: get_auth_request con 0x7fe2ec06e060 auth_method 0 2024-03-25T18:06:20.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.846+0000 7fe3057fa640 10 client.4273.objecter ms_handle_connect 0x7fe2ec06e060 2024-03-25T18:06:20.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.848+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fe2fc054950 con 0x7fe3080a98f0 2024-03-25T18:06:20.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.848+0000 7fe3057fa640 10 client.4273.objecter ms_dispatch 0x7fe308000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:20.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.848+0000 7fe3057fa640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:20.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.848+0000 7fe3057fa640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:20.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:20.848+0000 7fe3057fa640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T18:06:21.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.112+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2fc050290 con 0x7fe3080a98f0 2024-03-25T18:06:21.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.112+0000 7fe3057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:21.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.112+0000 7fe3057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:21.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.364+0000 7fe30f16d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:21.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.364+0000 7fe30f16d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:21.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.364+0000 7fe30f16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:21.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.364+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3081dae00 con 0x7fe3080a98f0 2024-03-25T18:06:21.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.394+0000 7fe3057fa640 1 -- 172.21.15.84:0/756707912 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fe2fc052390 con 0x7fe3080a98f0 2024-03-25T18:06:21.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.394+0000 7fe3057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:21.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.394+0000 7fe3057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:21.394 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:21.394 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:21.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 10 librados: watch_flush enter 2024-03-25T18:06:21.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 10 librados: watch_flush exit 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 10 client.4273.objecter successfully canceled tick 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 >> v1:172.21.15.157:6800/1801259026 conn(0x7fe2ec06e060 legacy=0x7fe2ec070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 10 monclient: shutdown 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 >> v1:172.21.15.157:6789/0 conn(0x7fe3080a98f0 legacy=0x7fe308137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 shutdown_connections 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 >> 172.21.15.84:0/756707912 conn(0x7fe3080b7ea0 msgr2=0x7fe308074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 shutdown_connections 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 -- 172.21.15.84:0/756707912 wait complete. 2024-03-25T18:06:21.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:21.397+0000 7fe30f16d640 1 librados: shutdown 2024-03-25T18:06:21.409 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:28.215 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 10 monclient: get_monmap_and_config 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 10 monclient: build_initial_monmap 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 10 monclient: monmap: 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:28.322509+0000 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:28.322509+0000 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:28.322 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:28.323 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 1 Processor -- start 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 1 -- start start 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.322+0000 7f6cb93d6640 10 monclient: init 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb40705b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb41402d0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb4149ae0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb40718c0 con 0x7f6cb4149ae0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb40714c0 con 0x7f6cb40705b0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb4072b50 con 0x7f6cb41402d0 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:28.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.323+0000 7f6cb93d6640 10 monclient(hunting): authenticate will time out at 7032.621641s 2024-03-25T18:06:28.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.343+0000 7f6cb37fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6cb4149ae0 0x7f6cb414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47812/0 (socket says 172.21.15.84:47812) 2024-03-25T18:06:28.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.343+0000 7f6cb37fe640 1 -- 172.21.15.84:0/61173266 learned_addr learned my addr 172.21.15.84:0/61173266 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:28.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318997727 0 0) 0x7f6cb40714c0 con 0x7f6cb40705b0 2024-03-25T18:06:28.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:28.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:28.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:28.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:06:28.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.349+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c003520 con 0x7f6cb40705b0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3446611980 0 0) 0x7f6c9c003520 con 0x7f6cb40705b0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 >> v1:172.21.15.84:6790/0 conn(0x7f6cb41402d0 legacy=0x7f6cb4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 >> v1:172.21.15.84:6789/0 conn(0x7f6cb4149ae0 legacy=0x7f6cb414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb413dd10 con 0x7f6cb40705b0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:28.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ca4001690 con 0x7f6cb40705b0 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb93d6640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb93d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb1ffb640 10 monclient: dump: 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.358 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:28.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:28.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:28.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.357+0000 7f6cb93d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca40017f0 con 0x7f6cb40705b0 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 1 -- 172.21.15.84:0/61173266 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ca4001c00 con 0x7f6cb40705b0 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb93d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb93d6640 4 set_mon_vals no callback set 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:28.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.387+0000 7f6cb1ffb640 10 monclient: dump: 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:28.388 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 10 monclient: shutdown 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 1 -- 172.21.15.84:0/61173266 >> v1:172.21.15.157:6789/0 conn(0x7f6cb40705b0 legacy=0x7f6cb413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 1 -- 172.21.15.84:0/61173266 shutdown_connections 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 1 -- 172.21.15.84:0/61173266 >> 172.21.15.84:0/61173266 conn(0x7f6cb40b5d10 msgr2=0x7f6cb40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 1 -- 172.21.15.84:0/61173266 shutdown_connections 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.388+0000 7f6cb93d6640 1 -- 172.21.15.84:0/61173266 wait complete. 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.389+0000 7f6cb93d6640 10 monclient: build_initial_monmap 2024-03-25T18:06:28.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.389+0000 7f6cb93d6640 10 monclient: monmap: 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.389+0000 7f6cb93d6640 1 librados: starting msgr at 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.389+0000 7f6cb93d6640 1 librados: starting objecter 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 1 Processor -- start 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 1 -- start start 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 1 librados: setting wanted keys 2024-03-25T18:06:28.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 1 librados: calling monclient init 2024-03-25T18:06:28.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 10 monclient: init 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.390+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb40705b0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb41402d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb4149ae0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb41d5540 con 0x7f6cb4149ae0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb41d60e0 con 0x7f6cb41402d0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb41d6270 con 0x7f6cb40705b0 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:28.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.391+0000 7f6cb93d6640 10 monclient(hunting): authenticate will time out at 7032.689061s 2024-03-25T18:06:28.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.402+0000 7f6cb27fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6cb41402d0 0x7f6cb4133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:40410/0 (socket says 172.21.15.84:40410) 2024-03-25T18:06:28.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.402+0000 7f6cb27fc640 1 -- 172.21.15.84:0/3934234150 learned_addr learned my addr 172.21.15.84:0/3934234150 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:28.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.405+0000 7f6c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6cb40705b0 2024-03-25T18:06:28.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.405+0000 7f6c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:28.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.406+0000 7f6c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6cb41402d0 2024-03-25T18:06:28.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.406+0000 7f6c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:28.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.412+0000 7f6c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f6cb4149ae0 2024-03-25T18:06:28.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.412+0000 7f6c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430047599 0 0) 0x7f6cb41d60e0 con 0x7f6cb41402d0 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:06:28.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.420+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca0004eb0 con 0x7f6cb41402d0 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599027133 0 0) 0x7f6cb41d6270 con 0x7f6cb40705b0 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:06:28.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.426+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb41d60e0 con 0x7f6cb40705b0 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484909440 0 0) 0x7f6cb41d5540 con 0x7f6cb4149ae0 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:06:28.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.428+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb41d6270 con 0x7f6cb4149ae0 2024-03-25T18:06:28.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460275256 0 0) 0x7f6ca0004eb0 con 0x7f6cb41402d0 2024-03-25T18:06:28.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb41d5540 con 0x7f6cb41402d0 2024-03-25T18:06:28.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ca80019d0 con 0x7f6cb41402d0 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.437+0000 7f6c9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:28.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3508049746 0 0) 0x7f6cb41d60e0 con 0x7f6cb40705b0 2024-03-25T18:06:28.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ca0004eb0 con 0x7f6cb40705b0 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ca4001b40 con 0x7f6cb40705b0 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.443+0000 7f6c9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:28.444 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3049014736 0 0) 0x7f6cb41d5540 con 0x7f6cb41402d0 2024-03-25T18:06:28.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 >> v1:172.21.15.84:6790/0 conn(0x7f6cb40705b0 legacy=0x7f6cb41d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 >> v1:172.21.15.84:6789/0 conn(0x7f6cb4149ae0 legacy=0x7f6cb41dfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:28.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 1 monclient: found mon.b 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb4133150 con 0x7f6cb41402d0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6c9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 10 monclient: _renew_subs 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb41333a0 con 0x7f6cb41402d0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 10 monclient: _renew_subs 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:28.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb41362a0 con 0x7f6cb41402d0 2024-03-25T18:06:28.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.457+0000 7f6cb93d6640 1 librados: init done 2024-03-25T18:06:28.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.458+0000 7f6cb93d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:28.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.458+0000 7f6cb93d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:28.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.458+0000 7f6cb93d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:28.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.458+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb40627b0 con 0x7f6cb41402d0 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca8003610 con 0x7f6cb41402d0 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6ca80039a0 con 0x7f6cb41402d0 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6c9b7fe640 10 monclient: dump: 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:28.752 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.752+0000 7f6cb17fa640 4 set_mon_vals no callback set 2024-03-25T18:06:28.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.753+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f6ca8003bc0 con 0x7f6cb41402d0 2024-03-25T18:06:28.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.753+0000 7f6cb2ffd640 10 monclient: get_auth_request con 0x7f6ca006df70 auth_method 0 2024-03-25T18:06:28.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.754+0000 7f6c9b7fe640 10 client.4279.objecter ms_handle_connect 0x7f6ca006df70 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f6ca80547e0 con 0x7f6cb41402d0 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 10 client.4279.objecter ms_dispatch 0x7f6cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ca80502f0 con 0x7f6cb41402d0 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:28.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:28.759+0000 7f6c9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:29.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.017+0000 7f6cb93d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:29.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.017+0000 7f6cb93d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:29.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.017+0000 7f6cb93d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:29.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.017+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb4069f40 con 0x7f6cb41402d0 2024-03-25T18:06:29.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.055+0000 7f6c9b7fe640 1 -- 172.21.15.84:0/3934234150 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f6ca8052220 con 0x7f6cb41402d0 2024-03-25T18:06:29.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.055+0000 7f6c9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:29.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.055+0000 7f6c9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:29.055 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:29.055 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 10 librados: watch_flush enter 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 10 librados: watch_flush exit 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 10 client.4279.objecter successfully canceled tick 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 >> v1:172.21.15.157:6800/1801259026 conn(0x7f6ca006df70 legacy=0x7f6ca0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 10 monclient: shutdown 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 >> v1:172.21.15.157:6789/0 conn(0x7f6cb41402d0 legacy=0x7f6cb4133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:29.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 shutdown_connections 2024-03-25T18:06:29.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.057+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 >> 172.21.15.84:0/3934234150 conn(0x7f6cb40b5d10 msgr2=0x7f6cb413b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:29.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.058+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 shutdown_connections 2024-03-25T18:06:29.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.058+0000 7f6cb93d6640 1 -- 172.21.15.84:0/3934234150 wait complete. 2024-03-25T18:06:29.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:29.058+0000 7f6cb93d6640 1 librados: shutdown 2024-03-25T18:06:29.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:35.876 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.987+0000 7f248ba3c640 10 monclient: get_monmap_and_config 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.987+0000 7f248ba3c640 10 monclient: build_initial_monmap 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.987+0000 7f248ba3c640 10 monclient: monmap: 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:35.988952+0000 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:35.988952+0000 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:35.988 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.988+0000 7f248ba3c640 1 Processor -- start 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.988+0000 7f248ba3c640 1 -- start start 2024-03-25T18:06:35.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.988+0000 7f248ba3c640 10 monclient: init 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-b con 0x7f2484061830 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-c con 0x7f24841400e0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-a con 0x7f24841499d0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484072600 con 0x7f24841499d0 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484140c80 con 0x7f24841400e0 2024-03-25T18:06:35.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484072830 con 0x7f2484061830 2024-03-25T18:06:35.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:35.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:35.989+0000 7f248ba3c640 10 monclient(hunting): authenticate will time out at 7040.288623s 2024-03-25T18:06:36.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.000+0000 7f24897d8640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f2484061830 0x7f248413d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59116/0 (socket says 172.21.15.84:59116) 2024-03-25T18:06:36.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.000+0000 7f24897d8640 1 -- 172.21.15.84:0/1522366829 learned_addr learned my addr 172.21.15.84:0/1522366829 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060644863 0 0) 0x7f2484072600 con 0x7f24841499d0 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:06:36.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.017+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2470003520 con 0x7f24841499d0 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665806115 0 0) 0x7f2484140c80 con 0x7f24841400e0 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:06:36.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.023+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2484072600 con 0x7f24841400e0 2024-03-25T18:06:36.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281068582 0 0) 0x7f2484072600 con 0x7f24841400e0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 >> v1:172.21.15.84:6790/0 conn(0x7f2484061830 legacy=0x7f248413d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 >> v1:172.21.15.84:6789/0 conn(0x7f24841499d0 legacy=0x7f248414bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 monclient: found mon.noname-c 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2484072b10 con 0x7f24841400e0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: _finish_auth 0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2474001670 con 0x7f24841400e0 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f248ba3c640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f248ba3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:36.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f247bfff640 10 monclient: dump: 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:36.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.031+0000 7f248ba3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24740017d0 con 0x7f24841400e0 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 1 -- 172.21.15.84:0/1522366829 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2474001ba0 con 0x7f24841400e0 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:36.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f247bfff640 10 monclient: dump: 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f248ba3c640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:36.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.037+0000 7f248ba3c640 4 set_mon_vals no callback set 2024-03-25T18:06:36.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 10 monclient: shutdown 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 1 -- 172.21.15.84:0/1522366829 >> v1:172.21.15.157:6789/0 conn(0x7f24841400e0 legacy=0x7f2484140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 1 -- 172.21.15.84:0/1522366829 shutdown_connections 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 1 -- 172.21.15.84:0/1522366829 >> 172.21.15.84:0/1522366829 conn(0x7f24840b0ef0 msgr2=0x7f24840b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 1 -- 172.21.15.84:0/1522366829 shutdown_connections 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.038+0000 7f248ba3c640 1 -- 172.21.15.84:0/1522366829 wait complete. 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.039+0000 7f248ba3c640 10 monclient: build_initial_monmap 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.039+0000 7f248ba3c640 10 monclient: monmap: 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:36.041 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.039+0000 7f248ba3c640 1 librados: starting msgr at 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.039+0000 7f248ba3c640 1 librados: starting objecter 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.039+0000 7f248ba3c640 1 Processor -- start 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 1 -- start start 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 1 librados: setting wanted keys 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 1 librados: calling monclient init 2024-03-25T18:06:36.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 10 monclient: init 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-c con 0x7f2484061830 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.040+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-b con 0x7f24841400e0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 10 monclient(hunting): picked mon.noname-a con 0x7f24841499d0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:36.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484148ac0 con 0x7f24841499d0 2024-03-25T18:06:36.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484137220 con 0x7f24841400e0 2024-03-25T18:06:36.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484147ec0 con 0x7f2484061830 2024-03-25T18:06:36.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:36.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.041+0000 7f248ba3c640 10 monclient(hunting): authenticate will time out at 7040.340032s 2024-03-25T18:06:36.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.047+0000 7f2488fd7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f24841400e0 0x7f2484136400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34660/0 (socket says 172.21.15.84:34660) 2024-03-25T18:06:36.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.047+0000 7f2488fd7640 1 -- 172.21.15.84:0/29887633 learned_addr learned my addr 172.21.15.84:0/29887633 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:36.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.051+0000 7f2479ffb640 10 client.?.objecter ms_handle_connect 0x7f24841400e0 2024-03-25T18:06:36.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.051+0000 7f2479ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:36.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.056+0000 7f2479ffb640 10 client.?.objecter ms_handle_connect 0x7f24841499d0 2024-03-25T18:06:36.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.056+0000 7f2479ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943664170 0 0) 0x7f2484137220 con 0x7f24841400e0 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:06:36.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.057+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2468004eb0 con 0x7f24841400e0 2024-03-25T18:06:36.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 client.?.objecter ms_handle_connect 0x7f2484061830 2024-03-25T18:06:36.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:36.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585232983 0 0) 0x7f2484148ac0 con 0x7f24841499d0 2024-03-25T18:06:36.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:36.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:36.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:36.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:06:36.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.066+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2484137220 con 0x7f24841499d0 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885274200 0 0) 0x7f2468004eb0 con 0x7f24841400e0 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2484148ac0 con 0x7f24841400e0 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f24740015f0 con 0x7f24841400e0 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 10 monclient(hunting): dump: 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.070 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:36.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.069+0000 7f2479ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:36.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1160153040 0 0) 0x7f2484148ac0 con 0x7f24841400e0 2024-03-25T18:06:36.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 >> v1:172.21.15.84:6790/0 conn(0x7f2484061830 legacy=0x7f2484148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 >> v1:172.21.15.84:6789/0 conn(0x7f24841499d0 legacy=0x7f2484136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 1 monclient: found mon.b 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24841481a0 con 0x7f24841400e0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f2479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 10 monclient: _renew_subs 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2484149660 con 0x7f24841400e0 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 10 monclient: _renew_subs 2024-03-25T18:06:36.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24841487a0 con 0x7f24841400e0 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.077+0000 7f248ba3c640 1 librados: init done 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.078+0000 7f248ba3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.078+0000 7f248ba3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.078+0000 7f248ba3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.078+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2484148c70 con 0x7f24841400e0 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24740038a0 con 0x7f24841400e0 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2474003a90 con 0x7f24841400e0 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:36.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f2479ffb640 10 monclient: dump: 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:36.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.349+0000 7f247b7fe640 4 set_mon_vals no callback set 2024-03-25T18:06:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.350+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2474003c70 con 0x7f24841400e0 2024-03-25T18:06:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.350+0000 7f24897d8640 10 monclient: get_auth_request con 0x7f246806dcc0 auth_method 0 2024-03-25T18:06:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.351+0000 7f2479ffb640 10 client.4285.objecter ms_handle_connect 0x7f246806dcc0 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.357+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f24740547c0 con 0x7f24841400e0 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.357+0000 7f2479ffb640 10 client.4285.objecter ms_dispatch 0x7f2484000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.357+0000 7f2479ffb640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.357+0000 7f2479ffb640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.358+0000 7f2479ffb640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T18:06:36.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.358+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24740502d0 con 0x7f24841400e0 2024-03-25T18:06:36.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.359+0000 7f2479ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:36.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.359+0000 7f2479ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:36.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.611+0000 7f248ba3c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:36.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.611+0000 7f248ba3c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:36.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.611+0000 7f248ba3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:36.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.611+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24840627b0 con 0x7f24841400e0 2024-03-25T18:06:36.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.756+0000 7f2479ffb640 1 -- 172.21.15.84:0/29887633 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f2474052200 con 0x7f24841400e0 2024-03-25T18:06:36.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.756+0000 7f2479ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:36.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.756+0000 7f2479ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:36.758 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:36.758 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:36.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 10 librados: watch_flush enter 2024-03-25T18:06:36.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 10 librados: watch_flush exit 2024-03-25T18:06:36.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T18:06:36.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 10 client.4285.objecter successfully canceled tick 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 >> v1:172.21.15.157:6800/1801259026 conn(0x7f246806dcc0 legacy=0x7f2468070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 10 monclient: shutdown 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 >> v1:172.21.15.157:6789/0 conn(0x7f24841400e0 legacy=0x7f2484136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 shutdown_connections 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 >> 172.21.15.84:0/29887633 conn(0x7f24840b0ef0 msgr2=0x7f24840af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 shutdown_connections 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 -- 172.21.15.84:0/29887633 wait complete. 2024-03-25T18:06:36.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:36.759+0000 7f248ba3c640 1 librados: shutdown 2024-03-25T18:06:36.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:37.487 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:34.307 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.495 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:34.314 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.544 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:34.328 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.547 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:34.366 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.547 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:34.331 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.558 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:34.378 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.559 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:34.379 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.607 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:34.391 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:37.609 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:34.428 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:43.578 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.691+0000 7fd9514ae640 10 monclient: get_monmap_and_config 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.691+0000 7fd9514ae640 10 monclient: build_initial_monmap 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.691+0000 7fd9514ae640 10 monclient: monmap: 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:43.692677+0000 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:43.692677+0000 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:43.692 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:43.693 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:43.693 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:43.693 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 1 Processor -- start 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 1 -- start start 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient: init 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd94c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd94c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd94c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c142520 con 0x7fd94c1430c0 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c0b28a0 con 0x7fd94c0a98f0 2024-03-25T18:06:43.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.692+0000 7fd9514ae640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c0b2a30 con 0x7fd94c13f710 2024-03-25T18:06:43.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.693+0000 7fd9514ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:43.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.693+0000 7fd9514ae640 10 monclient(hunting): authenticate will time out at 7047.991915s 2024-03-25T18:06:43.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.702+0000 7fd94affd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd94c0a98f0 0x7fd94c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56282/0 (socket says 172.21.15.84:56282) 2024-03-25T18:06:43.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.702+0000 7fd94affd640 1 -- 172.21.15.84:0/1002957756 learned_addr learned my addr 172.21.15.84:0/1002957756 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.730+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573617632 0 0) 0x7fd94c142520 con 0x7fd94c1430c0 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.731+0000 7fd94a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.731+0000 7fd94a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.731+0000 7fd94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.731+0000 7fd94a7fc640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:06:43.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.731+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938003520 con 0x7fd94c1430c0 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.732+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59626154 0 0) 0x7fd94c0b28a0 con 0x7fd94c0a98f0 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.732+0000 7fd94a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.732+0000 7fd94a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.732+0000 7fd94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.732+0000 7fd94a7fc640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c142520 con 0x7fd94c0a98f0 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895680978 0 0) 0x7fd94c0b2a30 con 0x7fd94c13f710 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:06:43.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.733+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c0b28a0 con 0x7fd94c13f710 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108175026 0 0) 0x7fd94c142520 con 0x7fd94c0a98f0 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 >> v1:172.21.15.84:6790/0 conn(0x7fd94c13f710 legacy=0x7fd94c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 >> v1:172.21.15.84:6789/0 conn(0x7fd94c1430c0 legacy=0x7fd94c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c0b44c0 con 0x7fd94c0a98f0 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:43.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd93c001670 con 0x7fd94c0a98f0 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd94a7fc640 10 monclient: dump: 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:43.744 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:43.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd9514ae640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:06:43.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.742+0000 7fd9514ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:43.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.768+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd93c0017d0 con 0x7fd94c0a98f0 2024-03-25T18:06:43.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.768+0000 7fd94a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.768+0000 7fd94a7fc640 1 -- 172.21.15.84:0/1002957756 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd93c001be0 con 0x7fd94c0a98f0 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.768+0000 7fd9514ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.768+0000 7fd9514ae640 4 set_mon_vals no callback set 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd94a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd94a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd94a7fc640 10 monclient: dump: 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:43.770 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:43.771 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:43.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 10 monclient: shutdown 2024-03-25T18:06:43.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 1 -- 172.21.15.84:0/1002957756 >> v1:172.21.15.157:6789/0 conn(0x7fd94c0a98f0 legacy=0x7fd94c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 1 -- 172.21.15.84:0/1002957756 shutdown_connections 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 1 -- 172.21.15.84:0/1002957756 >> 172.21.15.84:0/1002957756 conn(0x7fd94c0b7ea0 msgr2=0x7fd94c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 1 -- 172.21.15.84:0/1002957756 shutdown_connections 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.769+0000 7fd9514ae640 1 -- 172.21.15.84:0/1002957756 wait complete. 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.770+0000 7fd9514ae640 10 monclient: build_initial_monmap 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.770+0000 7fd9514ae640 10 monclient: monmap: 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:43.773 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.770+0000 7fd9514ae640 1 librados: starting msgr at 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.770+0000 7fd9514ae640 1 librados: starting objecter 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 Processor -- start 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 -- start start 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 librados: setting wanted keys 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 librados: calling monclient init 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient: init 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd94c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd94c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:43.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd94c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:43.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c13b180 con 0x7fd94c0a98f0 2024-03-25T18:06:43.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c1db890 con 0x7fd94c13f710 2024-03-25T18:06:43.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c0b7b90 con 0x7fd94c1430c0 2024-03-25T18:06:43.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:43.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.771+0000 7fd9514ae640 10 monclient(hunting): authenticate will time out at 7048.070841s 2024-03-25T18:06:43.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.776+0000 7fd943fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd94c13f710 0x7fd94c1daa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56350/0 (socket says 172.21.15.84:56350) 2024-03-25T18:06:43.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.776+0000 7fd943fff640 1 -- 172.21.15.84:0/3992551951 learned_addr learned my addr 172.21.15.84:0/3992551951 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:43.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.777+0000 7fd9437fe640 10 client.?.objecter ms_handle_connect 0x7fd94c0a98f0 2024-03-25T18:06:43.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.777+0000 7fd9437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:43.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.782+0000 7fd9437fe640 10 client.?.objecter ms_handle_connect 0x7fd94c13f710 2024-03-25T18:06:43.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.782+0000 7fd9437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:43.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710217732 0 0) 0x7fd94c1db890 con 0x7fd94c13f710 2024-03-25T18:06:43.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:43.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:43.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:43.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:06:43.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.793+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c004eb0 con 0x7fd94c13f710 2024-03-25T18:06:43.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732399706 0 0) 0x7fd94c13b180 con 0x7fd94c0a98f0 2024-03-25T18:06:43.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:43.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:43.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:43.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:06:43.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.796+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c1db890 con 0x7fd94c0a98f0 2024-03-25T18:06:43.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.803+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 120371986 0 0) 0x7fd92c004eb0 con 0x7fd94c13f710 2024-03-25T18:06:43.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.803+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd94c13b180 con 0x7fd94c13f710 2024-03-25T18:06:43.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.804+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd934001ad0 con 0x7fd94c13f710 2024-03-25T18:06:43.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.804+0000 7fd9437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:43.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.804+0000 7fd9437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:43.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.804+0000 7fd9437fe640 10 monclient(hunting): dump: 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:43.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.804+0000 7fd9437fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:43.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.806+0000 7fd9437fe640 10 client.?.objecter ms_handle_connect 0x7fd94c1430c0 2024-03-25T18:06:43.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.806+0000 7fd9437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:43.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 910709549 0 0) 0x7fd94c13b180 con 0x7fd94c13f710 2024-03-25T18:06:43.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 >> v1:172.21.15.84:6790/0 conn(0x7fd94c1430c0 legacy=0x7fd94c1db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 >> v1:172.21.15.84:6789/0 conn(0x7fd94c0a98f0 legacy=0x7fd94c0b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:43.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 1 monclient: found mon.b 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c13b3f0 con 0x7fd94c13f710 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 10 monclient: _renew_subs 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c13a1b0 con 0x7fd94c13f710 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 10 monclient: _renew_subs 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c13ba40 con 0x7fd94c13f710 2024-03-25T18:06:43.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.812+0000 7fd9514ae640 1 librados: init done 2024-03-25T18:06:43.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.813+0000 7fd9514ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:43.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.813+0000 7fd9514ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:43.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.813+0000 7fd9514ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:43.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:43.813+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94c0b3ef0 con 0x7fd94c13f710 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd934001c30 con 0x7fd94c13f710 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd934003ae0 con 0x7fd94c13f710 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd9437fe640 10 monclient: dump: 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:44.104 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.103+0000 7fd949ffb640 4 set_mon_vals no callback set 2024-03-25T18:06:44.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.104+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fd934003cc0 con 0x7fd94c13f710 2024-03-25T18:06:44.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.104+0000 7fd94affd640 10 monclient: get_auth_request con 0x7fd92c06dc80 auth_method 0 2024-03-25T18:06:44.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.105+0000 7fd9437fe640 10 client.4291.objecter ms_handle_connect 0x7fd92c06dc80 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fd9340549a0 con 0x7fd94c13f710 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 10 client.4291.objecter ms_dispatch 0x7fd94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9340502e0 con 0x7fd94c13f710 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:44.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.109+0000 7fd9437fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:44.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.333+0000 7fd9514ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:44.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.333+0000 7fd9514ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:44.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.333+0000 7fd9514ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:06:44.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.333+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd94c0627b0 con 0x7fd94c13f710 2024-03-25T18:06:44.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.371+0000 7fd9437fe640 1 -- 172.21.15.84:0/3992551951 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fd9340523e0 con 0x7fd94c13f710 2024-03-25T18:06:44.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.371+0000 7fd9437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:44.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.371+0000 7fd9437fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:44.373 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:44.373 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 10 librados: watch_flush enter 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 10 librados: watch_flush exit 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 10 client.4291.objecter successfully canceled tick 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 >> v1:172.21.15.157:6800/1801259026 conn(0x7fd92c06dc80 legacy=0x7fd92c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 10 monclient: shutdown 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 >> v1:172.21.15.157:6789/0 conn(0x7fd94c13f710 legacy=0x7fd94c1daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 shutdown_connections 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 >> 172.21.15.84:0/3992551951 conn(0x7fd94c0b7ea0 msgr2=0x7fd94c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 shutdown_connections 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 -- 172.21.15.84:0/3992551951 wait complete. 2024-03-25T18:06:44.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:44.374+0000 7fd9514ae640 1 librados: shutdown 2024-03-25T18:06:44.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:51.191 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.305+0000 7f77e5979640 10 monclient: get_monmap_and_config 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.305+0000 7f77e5979640 10 monclient: build_initial_monmap 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.305+0000 7f77e5979640 10 monclient: monmap: 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:51.307206+0000 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:51.307206+0000 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:51.307 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:51.308 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.306+0000 7f77e5979640 1 Processor -- start 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.306+0000 7f77e5979640 1 -- start start 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.306+0000 7f77e5979640 10 monclient: init 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-a con 0x7f77e00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-b con 0x7f77e013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-c con 0x7f77e01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e0142520 con 0x7f77e00a98f0 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e00b28a0 con 0x7f77e01430c0 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e00b2a30 con 0x7f77e013f710 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:51.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.307+0000 7f77e5979640 10 monclient(hunting): authenticate will time out at 7055.606495s 2024-03-25T18:06:51.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.316+0000 7f77df7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f77e01430c0 0x7f77e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35604/0 (socket says 172.21.15.84:35604) 2024-03-25T18:06:51.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.316+0000 7f77df7fe640 1 -- 172.21.15.84:0/689988692 learned_addr learned my addr 172.21.15.84:0/689988692 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:51.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011664272 0 0) 0x7f77e00b2a30 con 0x7f77e013f710 2024-03-25T18:06:51.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:51.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:51.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:51.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:06:51.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.330+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8003520 con 0x7f77e013f710 2024-03-25T18:06:51.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.337+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109944590 0 0) 0x7f77e0142520 con 0x7f77e00a98f0 2024-03-25T18:06:51.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.338+0000 7f77ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:51.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.338+0000 7f77ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:51.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.338+0000 7f77ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:51.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.338+0000 7f77ddffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:06:51.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.338+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e00b2a30 con 0x7f77e00a98f0 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299638928 0 0) 0x7f77c8003520 con 0x7f77e013f710 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 >> v1:172.21.15.157:6789/0 conn(0x7f77e01430c0 legacy=0x7f77e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 >> v1:172.21.15.84:6789/0 conn(0x7f77e00a98f0 legacy=0x7f77e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 monclient: found mon.noname-b 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00b44c0 con 0x7f77e013f710 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:51.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f77d4001690 con 0x7f77e013f710 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77ddffb640 10 monclient: dump: 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77e5979640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:06:51.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.340+0000 7f77e5979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77d40017f0 con 0x7f77e013f710 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 1 -- 172.21.15.84:0/689988692 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f77d4001c00 con 0x7f77e013f710 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 4 set_mon_vals no callback set 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77ddffb640 10 monclient: dump: 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:51.350 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:51.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 10 monclient: shutdown 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 1 -- 172.21.15.84:0/689988692 >> v1:172.21.15.84:6790/0 conn(0x7f77e013f710 legacy=0x7f77e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.349+0000 7f77e5979640 1 -- 172.21.15.84:0/689988692 shutdown_connections 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.350+0000 7f77e5979640 1 -- 172.21.15.84:0/689988692 >> 172.21.15.84:0/689988692 conn(0x7f77e00b7ea0 msgr2=0x7f77e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.350+0000 7f77e5979640 1 -- 172.21.15.84:0/689988692 shutdown_connections 2024-03-25T18:06:51.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.350+0000 7f77e5979640 1 -- 172.21.15.84:0/689988692 wait complete. 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.350+0000 7f77e5979640 10 monclient: build_initial_monmap 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.350+0000 7f77e5979640 10 monclient: monmap: 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 librados: starting msgr at 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 librados: starting objecter 2024-03-25T18:06:51.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 Processor -- start 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 -- start start 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 librados: setting wanted keys 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 librados: calling monclient init 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient: init 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-b con 0x7f77e00a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-c con 0x7f77e013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient(hunting): picked mon.noname-a con 0x7f77e01430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e00aba00 con 0x7f77e01430c0 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e01d7490 con 0x7f77e00a98f0 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e01d8360 con 0x7f77e013f710 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:51.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.351+0000 7f77e5979640 10 monclient(hunting): authenticate will time out at 7055.650854s 2024-03-25T18:06:51.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.361+0000 7f77de7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f77e013f710 0x7f77e01d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43852/0 (socket says 172.21.15.84:43852) 2024-03-25T18:06:51.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.361+0000 7f77de7fc640 1 -- 172.21.15.84:0/3522629726 learned_addr learned my addr 172.21.15.84:0/3522629726 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:51.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.364+0000 7f77c7fff640 10 client.?.objecter ms_handle_connect 0x7f77e013f710 2024-03-25T18:06:51.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.364+0000 7f77c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:51.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.366+0000 7f77c7fff640 10 client.?.objecter ms_handle_connect 0x7f77e01430c0 2024-03-25T18:06:51.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.366+0000 7f77c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:51.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716260348 0 0) 0x7f77e01d8360 con 0x7f77e013f710 2024-03-25T18:06:51.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:51.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:51.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:51.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:06:51.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.372+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77cc004eb0 con 0x7f77e013f710 2024-03-25T18:06:51.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 client.?.objecter ms_handle_connect 0x7f77e00a98f0 2024-03-25T18:06:51.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:51.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077485798 0 0) 0x7f77e00aba00 con 0x7f77e01430c0 2024-03-25T18:06:51.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:51.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:51.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:51.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:06:51.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.374+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e01d8360 con 0x7f77e01430c0 2024-03-25T18:06:51.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1741721189 0 0) 0x7f77cc004eb0 con 0x7f77e013f710 2024-03-25T18:06:51.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77e00aba00 con 0x7f77e013f710 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f77d4001bc0 con 0x7f77e013f710 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.379+0000 7f77c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2014201725 0 0) 0x7f77e00aba00 con 0x7f77e013f710 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 >> v1:172.21.15.157:6789/0 conn(0x7f77e00a98f0 legacy=0x7f77e01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 >> v1:172.21.15.84:6789/0 conn(0x7f77e01430c0 legacy=0x7f77e0133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 1 monclient: found mon.c 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00abc30 con 0x7f77e013f710 2024-03-25T18:06:51.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.386+0000 7f77c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 10 monclient: _renew_subs 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e0137940 con 0x7f77e013f710 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 10 monclient: _renew_subs 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e0135220 con 0x7f77e013f710 2024-03-25T18:06:51.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.387+0000 7f77e5979640 1 librados: init done 2024-03-25T18:06:51.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.388+0000 7f77e5979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:51.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.388+0000 7f77e5979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:51.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.388+0000 7f77e5979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.388+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e00627b0 con 0x7f77e013f710 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77d40011f0 con 0x7f77e013f710 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f77d4003b00 con 0x7f77e013f710 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77c7fff640 10 monclient: dump: 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:51.617 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:51.618 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:51.618 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:51.618 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:51.618 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:51.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.616+0000 7f77dd7fa640 4 set_mon_vals no callback set 2024-03-25T18:06:51.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.690+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f77d4003c60 con 0x7f77e013f710 2024-03-25T18:06:51.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.690+0000 7f77deffd640 10 monclient: get_auth_request con 0x7f77cc06dd50 auth_method 0 2024-03-25T18:06:51.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.691+0000 7f77c7fff640 10 client.4298.objecter ms_handle_connect 0x7f77cc06dd50 2024-03-25T18:06:51.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f77d40548d0 con 0x7f77e013f710 2024-03-25T18:06:51.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 10 client.4298.objecter ms_dispatch 0x7f77e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77d4050210 con 0x7f77e013f710 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:51.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.696+0000 7f77c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:51.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.911+0000 7f77e5979640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.911+0000 7f77e5979640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.911+0000 7f77e5979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:06:51.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.911+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77e01d7830 con 0x7f77e013f710 2024-03-25T18:06:51.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.939+0000 7f77c7fff640 1 -- 172.21.15.84:0/3522629726 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f77d4052310 con 0x7f77e013f710 2024-03-25T18:06:51.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.939+0000 7f77c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.939+0000 7f77c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:51.940 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:51.941 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.941+0000 7f77e5979640 10 librados: watch_flush enter 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.941+0000 7f77e5979640 10 librados: watch_flush exit 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.941+0000 7f77e5979640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.941+0000 7f77e5979640 10 client.4298.objecter successfully canceled tick 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 >> v1:172.21.15.157:6800/1801259026 conn(0x7f77cc06dd50 legacy=0x7f77cc070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 10 monclient: shutdown 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 >> v1:172.21.15.84:6790/0 conn(0x7f77e013f710 legacy=0x7f77e01d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 shutdown_connections 2024-03-25T18:06:51.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 >> 172.21.15.84:0/3522629726 conn(0x7f77e00b7ea0 msgr2=0x7f77e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:51.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 shutdown_connections 2024-03-25T18:06:51.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 -- 172.21.15.84:0/3522629726 wait complete. 2024-03-25T18:06:51.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:51.942+0000 7f77e5979640 1 librados: shutdown 2024-03-25T18:06:51.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:57.524 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.343 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.529 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.349 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.553 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.373 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.580 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:54.364 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.585 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:54.369 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.598 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:54.417 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.605 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:54.424 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.624 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:01:54.408 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:57.637 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:54.456 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:58.761 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.873+0000 7f9aa1430640 10 monclient: get_monmap_and_config 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.873+0000 7f9aa1430640 10 monclient: build_initial_monmap 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.873+0000 7f9aa1430640 10 monclient: monmap: 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:58.874 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:06:58.874929+0000 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:06:58.874929+0000 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.874+0000 7f9aa1430640 1 Processor -- start 2024-03-25T18:06:58.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.874+0000 7f9aa1430640 1 -- start start 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.874+0000 7f9aa1430640 10 monclient: init 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-b con 0x7f9a9c0b1ef0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-a con 0x7f9a9c1473a0 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9c150070 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c0b2570 con 0x7f9a9c1473a0 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c146c20 con 0x7f9a9c150070 2024-03-25T18:06:58.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c146e80 con 0x7f9a9c0b1ef0 2024-03-25T18:06:58.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:58.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.875+0000 7f9aa1430640 10 monclient(hunting): authenticate will time out at 7063.174510s 2024-03-25T18:06:58.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.881+0000 7f9a9b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9a9c150070 0x7f9a9c152460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:60428/0 (socket says 172.21.15.84:60428) 2024-03-25T18:06:58.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.881+0000 7f9a9b7fe640 1 -- 172.21.15.84:0/2738262661 learned_addr learned my addr 172.21.15.84:0/2738262661 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411489704 0 0) 0x7f9a9c146c20 con 0x7f9a9c150070 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:06:58.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.889+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88003520 con 0x7f9a9c150070 2024-03-25T18:06:58.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853007519 0 0) 0x7f9a88003520 con 0x7f9a9c150070 2024-03-25T18:06:58.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 >> v1:172.21.15.84:6790/0 conn(0x7f9a9c0b1ef0 legacy=0x7f9a9c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 >> v1:172.21.15.84:6789/0 conn(0x7f9a9c1473a0 legacy=0x7f9a9c143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c06e1b0 con 0x7f9a9c150070 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9a94001a10 con 0x7f9a9c150070 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9a9a7fc640 10 monclient: dump: 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:58.904 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9aa1430640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:06:58.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.902+0000 7f9aa1430640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a94001b70 con 0x7f9a9c150070 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 1 -- 172.21.15.84:0/2738262661 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9a94001fc0 con 0x7f9a9c150070 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9aa1430640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9aa1430640 4 set_mon_vals no callback set 2024-03-25T18:06:58.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.911+0000 7f9a9a7fc640 10 monclient: dump: 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:58.913 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 10 monclient: shutdown 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 1 -- 172.21.15.84:0/2738262661 >> v1:172.21.15.157:6789/0 conn(0x7f9a9c150070 legacy=0x7f9a9c152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 1 -- 172.21.15.84:0/2738262661 shutdown_connections 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 1 -- 172.21.15.84:0/2738262661 >> 172.21.15.84:0/2738262661 conn(0x7f9a9c0b7ea0 msgr2=0x7f9a9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 1 -- 172.21.15.84:0/2738262661 shutdown_connections 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.912+0000 7f9aa1430640 1 -- 172.21.15.84:0/2738262661 wait complete. 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.913+0000 7f9aa1430640 10 monclient: build_initial_monmap 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.913+0000 7f9aa1430640 10 monclient: monmap: 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:06:58.915 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.913+0000 7f9aa1430640 1 librados: starting msgr at 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.913+0000 7f9aa1430640 1 librados: starting objecter 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.913+0000 7f9aa1430640 1 Processor -- start 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 -- start start 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 librados: setting wanted keys 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 librados: calling monclient init 2024-03-25T18:06:58.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient: init 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9c0b1ef0 addr v1:172.21.15.84:6790/0 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-b con 0x7f9a9c1473a0 addr v1:172.21.15.157:6789/0 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient(hunting): picked mon.noname-a con 0x7f9a9c150070 addr v1:172.21.15.84:6789/0 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c1df360 con 0x7f9a9c150070 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c1dff00 con 0x7f9a9c1473a0 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c1445d0 con 0x7f9a9c0b1ef0 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:58.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.914+0000 7f9aa1430640 10 monclient(hunting): authenticate will time out at 7063.213673s 2024-03-25T18:06:58.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.922+0000 7f9a9affd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f9a9c0b1ef0 0x7f9a9c1df7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58278/0 (socket says 172.21.15.84:58278) 2024-03-25T18:06:58.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.922+0000 7f9a9affd640 1 -- 172.21.15.84:0/3524148522 learned_addr learned my addr 172.21.15.84:0/3524148522 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:06:58.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.925+0000 7f9a93fff640 10 client.?.objecter ms_handle_connect 0x7f9a9c0b1ef0 2024-03-25T18:06:58.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.925+0000 7f9a93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.930+0000 7f9a93fff640 10 client.?.objecter ms_handle_connect 0x7f9a9c150070 2024-03-25T18:06:58.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.930+0000 7f9a93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 client.?.objecter ms_handle_connect 0x7f9a9c1473a0 2024-03-25T18:06:58.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633763278 0 0) 0x7f9a9c1445d0 con 0x7f9a9c0b1ef0 2024-03-25T18:06:58.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:06:58.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.934+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c004eb0 con 0x7f9a9c0b1ef0 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069794763 0 0) 0x7f9a9c1df360 con 0x7f9a9c150070 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:06:58.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.941+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a9c1445d0 con 0x7f9a9c150070 2024-03-25T18:06:58.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468627722 0 0) 0x7f9a9c1dff00 con 0x7f9a9c1473a0 2024-03-25T18:06:58.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:06:58.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.946+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a9c1df360 con 0x7f9a9c1473a0 2024-03-25T18:06:58.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1482962062 0 0) 0x7f9a9c1445d0 con 0x7f9a9c150070 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a9c1dff00 con 0x7f9a9c150070 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9a94001e80 con 0x7f9a9c150070 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 10 monclient(hunting): dump: 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:58.952 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.951+0000 7f9a93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:58.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994908739 0 0) 0x7f9a9c1df360 con 0x7f9a9c1473a0 2024-03-25T18:06:58.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a9c1445d0 con 0x7f9a9c1473a0 2024-03-25T18:06:58.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9a84001830 con 0x7f9a9c1473a0 2024-03-25T18:06:58.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.957+0000 7f9a93fff640 10 monclient(hunting): dump: 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:58.962 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3720793569 0 0) 0x7f9a9c1dff00 con 0x7f9a9c150070 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 >> v1:172.21.15.84:6790/0 conn(0x7f9a9c0b1ef0 legacy=0x7f9a9c1df7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 >> v1:172.21.15.157:6789/0 conn(0x7f9a9c1473a0 legacy=0x7f9a9c070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 1 monclient: found mon.a 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c1deff0 con 0x7f9a9c150070 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9a93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 10 monclient: _renew_subs 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:06:58.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c1e7330 con 0x7f9a9c150070 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 10 monclient: _renew_subs 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c1e76a0 con 0x7f9a9c150070 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.961+0000 7f9aa1430640 1 librados: init done 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.962+0000 7f9aa1430640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.962+0000 7f9aa1430640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.962+0000 7f9aa1430640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:06:58.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:58.962+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c0627b0 con 0x7f9a9c150070 2024-03-25T18:06:59.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a94001fe0 con 0x7f9a9c150070 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9a94004fb0 con 0x7f9a9c150070 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a93fff640 10 monclient: dump: 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:06:59.268 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:06:59.269 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:06:59.269 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:06:59.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.266+0000 7f9a99ffb640 4 set_mon_vals no callback set 2024-03-25T18:06:59.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.267+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f9a9401af80 con 0x7f9a9c150070 2024-03-25T18:06:59.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.268+0000 7f9a9affd640 10 monclient: get_auth_request con 0x7f9a7c06df30 auth_method 0 2024-03-25T18:06:59.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.268+0000 7f9a93fff640 10 client.4302.objecter ms_handle_connect 0x7f9a7c06df30 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f9a94054a70 con 0x7f9a9c150070 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 10 client.4302.objecter ms_dispatch 0x7f9a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a94050580 con 0x7f9a9c150070 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:59.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.273+0000 7f9a93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:59.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.524+0000 7f9aa1430640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:59.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.524+0000 7f9aa1430640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:59.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.524+0000 7f9aa1430640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:06:59.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.524+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a9c142d90 con 0x7f9a9c150070 2024-03-25T18:06:59.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.550+0000 7f9a93fff640 1 -- 172.21.15.84:0/3524148522 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f9a940524b0 con 0x7f9a9c150070 2024-03-25T18:06:59.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.550+0000 7f9a93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:59.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.550+0000 7f9a93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:59.551 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:06:59.551 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:59.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 10 librados: watch_flush enter 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 10 librados: watch_flush exit 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 >> v1:172.21.15.157:6800/1801259026 conn(0x7f9a7c06df30 legacy=0x7f9a7c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 10 monclient: shutdown 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 >> v1:172.21.15.84:6789/0 conn(0x7f9a9c150070 legacy=0x7f9a9c070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 shutdown_connections 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.552+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 >> 172.21.15.84:0/3524148522 conn(0x7f9a9c0b7ea0 msgr2=0x7f9a9c1416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.553+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 shutdown_connections 2024-03-25T18:06:59.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.553+0000 7f9aa1430640 1 -- 172.21.15.84:0/3524148522 wait complete. 2024-03-25T18:06:59.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:06:59.553+0000 7f9aa1430640 1 librados: shutdown 2024-03-25T18:06:59.565 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:06.372 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:06.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.485+0000 7fca10f38640 10 monclient: get_monmap_and_config 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.485+0000 7fca10f38640 10 monclient: build_initial_monmap 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.485+0000 7fca10f38640 10 monclient: monmap: 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:06.487177+0000 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:06.487177+0000 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:06.487 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.486+0000 7fca10f38640 1 Processor -- start 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.486+0000 7fca10f38640 1 -- start start 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.486+0000 7fca10f38640 10 monclient: init 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c142520 con 0x7fca0c13f710 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b28a0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b2a30 con 0x7fca0c1430c0 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.487+0000 7fca10f38640 10 monclient(hunting): authenticate will time out at 7070.786580s 2024-03-25T18:07:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.500+0000 7fca0ad9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fca0c1430c0 0x7fca0c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34818/0 (socket says 172.21.15.84:34818) 2024-03-25T18:07:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.500+0000 7fca0ad9d640 1 -- 172.21.15.84:0/706649609 learned_addr learned my addr 172.21.15.84:0/706649609 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674362226 0 0) 0x7fca0c0b28a0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f4003520 con 0x7fca0c0a98f0 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687334854 0 0) 0x7fca0c0b2a30 con 0x7fca0c1430c0 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:07:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.510+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0b28a0 con 0x7fca0c1430c0 2024-03-25T18:07:06.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658177363 0 0) 0x7fca0c142520 con 0x7fca0c13f710 2024-03-25T18:07:06.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:07:06.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.517+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0b2a30 con 0x7fca0c13f710 2024-03-25T18:07:06.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139658356 0 0) 0x7fca0c0b28a0 con 0x7fca0c1430c0 2024-03-25T18:07:06.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 >> v1:172.21.15.157:6789/0 conn(0x7fca0c0a98f0 legacy=0x7fca0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 >> v1:172.21.15.84:6789/0 conn(0x7fca0c13f710 legacy=0x7fca0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: _finish_hunting 0 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 monclient: found mon.noname-b 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0b44c0 con 0x7fca0c1430c0 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: _finish_auth 0 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: _check_auth_tickets 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fca040019f0 con 0x7fca0c1430c0 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca0959a640 10 monclient: dump: 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.529 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca10f38640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:07:06.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.527+0000 7fca10f38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca04001b50 con 0x7fca0c1430c0 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 1 -- 172.21.15.84:0/706649609 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fca04001f60 con 0x7fca0c1430c0 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca10f38640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca10f38640 4 set_mon_vals no callback set 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.546+0000 7fca0959a640 10 monclient: dump: 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:06.548 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:06.549 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:06.549 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 10 monclient: shutdown 2024-03-25T18:07:06.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 1 -- 172.21.15.84:0/706649609 >> v1:172.21.15.84:6790/0 conn(0x7fca0c1430c0 legacy=0x7fca0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 1 -- 172.21.15.84:0/706649609 shutdown_connections 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 1 -- 172.21.15.84:0/706649609 >> 172.21.15.84:0/706649609 conn(0x7fca0c0b7ea0 msgr2=0x7fca0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 1 -- 172.21.15.84:0/706649609 shutdown_connections 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.547+0000 7fca10f38640 1 -- 172.21.15.84:0/706649609 wait complete. 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.548+0000 7fca10f38640 10 monclient: build_initial_monmap 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.548+0000 7fca10f38640 10 monclient: monmap: 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:06.551 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 librados: starting msgr at 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 librados: starting objecter 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 Processor -- start 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 -- start start 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 librados: setting wanted keys 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 librados: calling monclient init 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient: init 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:06.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c12fa20 con 0x7fca0c13f710 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c12fc70 con 0x7fca0c0a98f0 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c13c080 con 0x7fca0c1430c0 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:06.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.549+0000 7fca10f38640 10 monclient(hunting): authenticate will time out at 7070.848876s 2024-03-25T18:07:06.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.560+0000 7fca0a59c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fca0c0a98f0 0x7fca0c12e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:37110/0 (socket says 172.21.15.84:37110) 2024-03-25T18:07:06.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.561+0000 7fca0a59c640 1 -- 172.21.15.84:0/1427153357 learned_addr learned my addr 172.21.15.84:0/1427153357 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:06.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.562+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c0a98f0 2024-03-25T18:07:06.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.562+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.572+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c1430c0 2024-03-25T18:07:06.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.572+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050666715 0 0) 0x7fca0c12fc70 con 0x7fca0c0a98f0 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.573+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ec004eb0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.576+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c13f710 2024-03-25T18:07:06.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.576+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052867150 0 0) 0x7fca0c12fa20 con 0x7fca0c13f710 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:07:06.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.584+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c12fc70 con 0x7fca0c13f710 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836103927 0 0) 0x7fc9ec004eb0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c12fa20 con 0x7fca0c0a98f0 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc9fc001990 con 0x7fca0c0a98f0 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 10 monclient(hunting): dump: 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.595 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.594+0000 7fc9faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972192994 0 0) 0x7fca0c13c080 con 0x7fca0c1430c0 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:07:06.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.595+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ec004eb0 con 0x7fca0c1430c0 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446057457 0 0) 0x7fca0c12fc70 con 0x7fca0c13f710 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c13c080 con 0x7fca0c13f710 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fca000018b0 con 0x7fca0c13f710 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.606+0000 7fc9faffd640 10 monclient(hunting): dump: 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.607 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:06.608 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 755017271 0 0) 0x7fca0c12fa20 con 0x7fca0c0a98f0 2024-03-25T18:07:06.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 >> v1:172.21.15.84:6790/0 conn(0x7fca0c1430c0 legacy=0x7fca0c1307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 >> v1:172.21.15.84:6789/0 conn(0x7fca0c13f710 legacy=0x7fca0c130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 1 monclient: found mon.b 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c13cb20 con 0x7fca0c0a98f0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fc9faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 10 monclient: _renew_subs 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c139d20 con 0x7fca0c0a98f0 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 10 monclient: _renew_subs 2024-03-25T18:07:06.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c131090 con 0x7fca0c0a98f0 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.612+0000 7fca10f38640 1 librados: init done 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.613+0000 7fca10f38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.613+0000 7fca10f38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.613+0000 7fca10f38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:06.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.613+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c0b3ef0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9fc003610 con 0x7fca0c0a98f0 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc9fc0039e0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 10 monclient: dump: 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:06.900 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fca08d99640 4 set_mon_vals no callback set 2024-03-25T18:07:06.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.900+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fc9fc003c00 con 0x7fca0c0a98f0 2024-03-25T18:07:06.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.901+0000 7fca09d9b640 10 monclient: get_auth_request con 0x7fc9ec06dfb0 auth_method 0 2024-03-25T18:07:06.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.901+0000 7fc9faffd640 10 client.4309.objecter ms_handle_connect 0x7fc9ec06dfb0 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.906+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fc9fc0547c0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.906+0000 7fc9faffd640 10 client.4309.objecter ms_dispatch 0x7fca0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.906+0000 7fc9faffd640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.906+0000 7fc9faffd640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.906+0000 7fc9faffd640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.907+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9fc0502d0 con 0x7fca0c0a98f0 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.907+0000 7fc9faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:06.907+0000 7fc9faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:07.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.142+0000 7fca10f38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:07.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.142+0000 7fca10f38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:07.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.142+0000 7fca10f38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:07.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.142+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca0c131340 con 0x7fca0c0a98f0 2024-03-25T18:07:07.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.207+0000 7fc9faffd640 1 -- 172.21.15.84:0/1427153357 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fc9fc052200 con 0x7fca0c0a98f0 2024-03-25T18:07:07.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.207+0000 7fc9faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:07.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.207+0000 7fc9faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:07.207 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:07.207 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 10 librados: watch_flush enter 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 10 librados: watch_flush exit 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 10 client.4309.objecter successfully canceled tick 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 >> v1:172.21.15.157:6800/1801259026 conn(0x7fc9ec06dfb0 legacy=0x7fc9ec070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 10 monclient: shutdown 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 >> v1:172.21.15.157:6789/0 conn(0x7fca0c0a98f0 legacy=0x7fca0c12e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 shutdown_connections 2024-03-25T18:07:07.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 >> 172.21.15.84:0/1427153357 conn(0x7fca0c0b7ea0 msgr2=0x7fca0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:07.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 shutdown_connections 2024-03-25T18:07:07.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 -- 172.21.15.84:0/1427153357 wait complete. 2024-03-25T18:07:07.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:07.209+0000 7fca10f38640 1 librados: shutdown 2024-03-25T18:07:07.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:14.027 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 10 monclient: get_monmap_and_config 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 10 monclient: build_initial_monmap 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 10 monclient: monmap: 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:14.142430+0000 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:14.142430+0000 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:14.142 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:14.143 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 1 Processor -- start 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 1 -- start start 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.142+0000 7f68cc501640 10 monclient: init 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-c con 0x7f68c40a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-a con 0x7f68c413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-b con 0x7f68c41430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4142520 con 0x7f68c413f710 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40b28a0 con 0x7f68c40a98f0 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40b2a30 con 0x7f68c41430c0 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:14.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.143+0000 7f68cc501640 10 monclient(hunting): authenticate will time out at 7078.441531s 2024-03-25T18:07:14.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.152+0000 7f68ca29d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f68c40a98f0 0x7f68c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50414/0 (socket says 172.21.15.84:50414) 2024-03-25T18:07:14.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.152+0000 7f68ca29d640 1 -- 172.21.15.84:0/2909129390 learned_addr learned my addr 172.21.15.84:0/2909129390 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451802803 0 0) 0x7f68c40b2a30 con 0x7f68c41430c0 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 10 monclient(hunting): _negotiate 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:07:14.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.163+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b0003520 con 0x7f68c41430c0 2024-03-25T18:07:14.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3483586303 0 0) 0x7f68b0003520 con 0x7f68c41430c0 2024-03-25T18:07:14.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 >> v1:172.21.15.157:6789/0 conn(0x7f68c40a98f0 legacy=0x7f68c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 >> v1:172.21.15.84:6789/0 conn(0x7f68c413f710 legacy=0x7f68c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: _finish_hunting 0 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 monclient: found mon.noname-b 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40b44c0 con 0x7f68c41430c0 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: _finish_auth 0 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: _check_auth_tickets 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68bc001a10 con 0x7f68c41430c0 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68c929b640 10 monclient: dump: 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.172 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68cc501640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:07:14.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.171+0000 7f68cc501640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68bc001b70 con 0x7f68c41430c0 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 1 -- 172.21.15.84:0/2909129390 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68bc001f80 con 0x7f68c41430c0 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68c929b640 10 monclient: dump: 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.215 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68cc501640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68cc501640 4 set_mon_vals no callback set 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68cc501640 10 monclient: shutdown 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68cc501640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.215+0000 7f68cc501640 1 -- 172.21.15.84:0/2909129390 >> v1:172.21.15.84:6790/0 conn(0x7f68c41430c0 legacy=0x7f68c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 1 -- 172.21.15.84:0/2909129390 shutdown_connections 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 1 -- 172.21.15.84:0/2909129390 >> 172.21.15.84:0/2909129390 conn(0x7f68c40b7ea0 msgr2=0x7f68c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 1 -- 172.21.15.84:0/2909129390 shutdown_connections 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 1 -- 172.21.15.84:0/2909129390 wait complete. 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 10 monclient: build_initial_monmap 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.216+0000 7f68cc501640 10 monclient: monmap: 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:14.218 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 librados: starting msgr at 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 librados: starting objecter 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 Processor -- start 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 -- start start 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 librados: setting wanted keys 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 librados: calling monclient init 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient: init 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-b con 0x7f68c40a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-c con 0x7f68c413f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient(hunting): picked mon.noname-a con 0x7f68c41430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4136ec0 con 0x7f68c41430c0 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4136c20 con 0x7f68c40a98f0 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4137db0 con 0x7f68c413f710 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:14.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.217+0000 7f68cc501640 10 monclient(hunting): authenticate will time out at 7078.515837s 2024-03-25T18:07:14.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.223+0000 7f68c9a9c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f68c413f710 0x7f68c40b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53406/0 (socket says 172.21.15.84:53406) 2024-03-25T18:07:14.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.223+0000 7f68c9a9c640 1 -- 172.21.15.84:0/404254348 learned_addr learned my addr 172.21.15.84:0/404254348 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:14.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.225+0000 7f68baffd640 10 client.?.objecter ms_handle_connect 0x7f68c413f710 2024-03-25T18:07:14.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.225+0000 7f68baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405670598 0 0) 0x7f68c4137db0 con 0x7f68c413f710 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:07:14.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.231+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8004eb0 con 0x7f68c413f710 2024-03-25T18:07:14.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.234+0000 7f68baffd640 10 client.?.objecter ms_handle_connect 0x7f68c40a98f0 2024-03-25T18:07:14.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.234+0000 7f68baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:14.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.235+0000 7f68baffd640 10 client.?.objecter ms_handle_connect 0x7f68c41430c0 2024-03-25T18:07:14.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.235+0000 7f68baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110065105 0 0) 0x7f68c4136ec0 con 0x7f68c41430c0 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:07:14.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.249+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c4137db0 con 0x7f68c41430c0 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688043535 0 0) 0x7f68c4136c20 con 0x7f68c40a98f0 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:07:14.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.250+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c4136ec0 con 0x7f68c40a98f0 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4016218001 0 0) 0x7f68a8004eb0 con 0x7f68c413f710 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c4136c20 con 0x7f68c413f710 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68b4001a30 con 0x7f68c413f710 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 10 monclient(hunting): dump: 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.265 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.265+0000 7f68baffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:14.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3508524159 0 0) 0x7f68c4136ec0 con 0x7f68c40a98f0 2024-03-25T18:07:14.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a8004eb0 con 0x7f68c40a98f0 2024-03-25T18:07:14.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68c0001860 con 0x7f68c40a98f0 2024-03-25T18:07:14.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.273+0000 7f68baffd640 10 monclient(hunting): dump: 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.274 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769497302 0 0) 0x7f68c4137db0 con 0x7f68c41430c0 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c4136ec0 con 0x7f68c41430c0 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68bc001e80 con 0x7f68c41430c0 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.274+0000 7f68baffd640 10 monclient(hunting): dump: 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.275 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.276 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3069920171 0 0) 0x7f68c4136c20 con 0x7f68c413f710 2024-03-25T18:07:14.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 >> v1:172.21.15.157:6789/0 conn(0x7f68c40a98f0 legacy=0x7f68c41376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 >> v1:172.21.15.84:6789/0 conn(0x7f68c41430c0 legacy=0x7f68c41e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 1 monclient: found mon.c 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4138b50 con 0x7f68c413f710 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 5 monclient: authenticate success, global_id 4316 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 10 monclient: _renew_subs 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c4139830 con 0x7f68c413f710 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 10 monclient: _renew_subs 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c41369b0 con 0x7f68c413f710 2024-03-25T18:07:14.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.278+0000 7f68cc501640 1 librados: init done 2024-03-25T18:07:14.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.279+0000 7f68cc501640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:14.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.279+0000 7f68cc501640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:14.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.279+0000 7f68cc501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.279+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c4139a20 con 0x7f68c413f710 2024-03-25T18:07:14.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68b4001c70 con 0x7f68c413f710 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f68b40038c0 con 0x7f68c413f710 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68baffd640 10 monclient: dump: 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:14.580 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:14.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.579+0000 7f68c8a9a640 4 set_mon_vals no callback set 2024-03-25T18:07:14.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.580+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f68b4003aa0 con 0x7f68c413f710 2024-03-25T18:07:14.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.581+0000 7f68ca29d640 10 monclient: get_auth_request con 0x7f68a806e0f0 auth_method 0 2024-03-25T18:07:14.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.581+0000 7f68baffd640 10 client.4316.objecter ms_handle_connect 0x7f68a806e0f0 2024-03-25T18:07:14.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f68b40546f0 con 0x7f68c413f710 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 10 client.4316.objecter ms_dispatch 0x7f68c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68b4050200 con 0x7f68c413f710 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:14.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.588+0000 7f68baffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:14.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.830+0000 7f68cc501640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.830+0000 7f68cc501640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.830+0000 7f68cc501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:14.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.830+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68c414f320 con 0x7f68c413f710 2024-03-25T18:07:14.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.873+0000 7f68baffd640 1 -- 172.21.15.84:0/404254348 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f68b4052130 con 0x7f68c413f710 2024-03-25T18:07:14.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.873+0000 7f68baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.873+0000 7f68baffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:14.873 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:14.874 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.875+0000 7f68cc501640 10 librados: watch_flush enter 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.875+0000 7f68cc501640 10 librados: watch_flush exit 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 10 client.4316.objecter successfully canceled tick 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 >> v1:172.21.15.157:6800/1801259026 conn(0x7f68a806e0f0 legacy=0x7f68a80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 10 monclient: shutdown 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 >> v1:172.21.15.84:6790/0 conn(0x7f68c413f710 legacy=0x7f68c40b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 shutdown_connections 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 >> 172.21.15.84:0/404254348 conn(0x7f68c40b7ea0 msgr2=0x7f68c413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 shutdown_connections 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 -- 172.21.15.84:0/404254348 wait complete. 2024-03-25T18:07:14.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:14.876+0000 7f68cc501640 1 librados: shutdown 2024-03-25T18:07:14.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:17.534 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:14.353 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.584 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:14.404 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.615 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:14.434 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.636 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:14.420 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.643 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:14.427 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.649 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:14.468 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.663 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:14.483 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.670 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:14.454 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:17.691 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:14.511 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:21.695 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:21.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.793+0000 7fe0cfa33640 10 monclient: get_monmap_and_config 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.793+0000 7fe0cfa33640 10 monclient: build_initial_monmap 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.793+0000 7fe0cfa33640 10 monclient: monmap: 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:21.794308+0000 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:21.794308+0000 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:21.794 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.794+0000 7fe0cfa33640 1 Processor -- start 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.794+0000 7fe0cfa33640 1 -- start start 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.794+0000 7fe0cfa33640 10 monclient: init 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c80a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c81430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c8142520 con 0x7fe0c81430c0 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80b28a0 con 0x7fe0c80a98f0 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80b2a30 con 0x7fe0c813f710 2024-03-25T18:07:21.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.795+0000 7fe0cfa33640 10 monclient(hunting): authenticate will time out at 7086.093449s 2024-03-25T18:07:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.796+0000 7fe0cd7cf640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe0c80a98f0 0x7fe0c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:32976/0 (socket says 172.21.15.84:32976) 2024-03-25T18:07:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.796+0000 7fe0cd7cf640 1 -- 172.21.15.84:0/2968868486 learned_addr learned my addr 172.21.15.84:0/2968868486 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377733649 0 0) 0x7fe0c8142520 con 0x7fe0c81430c0 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:07:21.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.825+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b4003520 con 0x7fe0c81430c0 2024-03-25T18:07:21.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261376127 0 0) 0x7fe0c80b28a0 con 0x7fe0c80a98f0 2024-03-25T18:07:21.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:07:21.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.832+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c8142520 con 0x7fe0c80a98f0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.837+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630148446 0 0) 0x7fe0b4003520 con 0x7fe0c81430c0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.837+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 >> v1:172.21.15.84:6790/0 conn(0x7fe0c813f710 legacy=0x7fe0c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.837+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 >> v1:172.21.15.157:6789/0 conn(0x7fe0c80a98f0 legacy=0x7fe0c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 1 monclient: found mon.noname-a 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80b44c0 con 0x7fe0c81430c0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: _finish_auth 0 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:21.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0cfa33640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe0c0001a10 con 0x7fe0c81430c0 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0bffff640 10 monclient: dump: 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.838+0000 7fe0cfa33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c0001b70 con 0x7fe0c81430c0 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 1 -- 172.21.15.84:0/2968868486 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe0c0001f80 con 0x7fe0c81430c0 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0cfa33640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0cfa33640 4 set_mon_vals no callback set 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0bffff640 10 monclient: dump: 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.851 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.852 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:21.852 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:21.852 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:21.852 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:21.852 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0cfa33640 10 monclient: shutdown 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0cfa33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.851+0000 7fe0cfa33640 1 -- 172.21.15.84:0/2968868486 >> v1:172.21.15.84:6789/0 conn(0x7fe0c81430c0 legacy=0x7fe0c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.852+0000 7fe0cfa33640 1 -- 172.21.15.84:0/2968868486 shutdown_connections 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.852+0000 7fe0cfa33640 1 -- 172.21.15.84:0/2968868486 >> 172.21.15.84:0/2968868486 conn(0x7fe0c80b7ea0 msgr2=0x7fe0c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.852+0000 7fe0cfa33640 1 -- 172.21.15.84:0/2968868486 shutdown_connections 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.852+0000 7fe0cfa33640 1 -- 172.21.15.84:0/2968868486 wait complete. 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 10 monclient: build_initial_monmap 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 10 monclient: monmap: 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:21.853 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:21.854 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 librados: starting msgr at 2024-03-25T18:07:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 librados: starting objecter 2024-03-25T18:07:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 Processor -- start 2024-03-25T18:07:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 -- start start 2024-03-25T18:07:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 librados: setting wanted keys 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 1 librados: calling monclient init 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.853+0000 7fe0cfa33640 10 monclient: init 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c80a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c813f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c81430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c8136c50 con 0x7fe0c80a98f0 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c8136ec0 con 0x7fe0c813f710 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c81317f0 con 0x7fe0c81430c0 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:21.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.854+0000 7fe0cfa33640 10 monclient(hunting): authenticate will time out at 7086.152818s 2024-03-25T18:07:21.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.860+0000 7fe0cd7cf640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe0c80a98f0 0x7fe0c81310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44302/0 (socket says 172.21.15.84:44302) 2024-03-25T18:07:21.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.860+0000 7fe0cd7cf640 1 -- 172.21.15.84:0/1148390145 learned_addr learned my addr 172.21.15.84:0/1148390145 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.862+0000 7fe0bdffb640 10 client.?.objecter ms_handle_connect 0x7fe0c80a98f0 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.862+0000 7fe0bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.866+0000 7fe0bdffb640 10 client.?.objecter ms_handle_connect 0x7fe0c81430c0 2024-03-25T18:07:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.866+0000 7fe0bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706233816 0 0) 0x7fe0c8136c50 con 0x7fe0c80a98f0 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:07:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.867+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac004eb0 con 0x7fe0c80a98f0 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953891361 0 0) 0x7fe0c81317f0 con 0x7fe0c81430c0 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:07:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.872+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c8136c50 con 0x7fe0c81430c0 2024-03-25T18:07:21.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.876+0000 7fe0bdffb640 10 client.?.objecter ms_handle_connect 0x7fe0c813f710 2024-03-25T18:07:21.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.876+0000 7fe0bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177221371 0 0) 0x7fe0c8136ec0 con 0x7fe0c813f710 2024-03-25T18:07:21.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:07:21.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.894+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c81317f0 con 0x7fe0c813f710 2024-03-25T18:07:21.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785980480 0 0) 0x7fe0c8136c50 con 0x7fe0c81430c0 2024-03-25T18:07:21.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0c8136ec0 con 0x7fe0c81430c0 2024-03-25T18:07:21.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe0c0001e80 con 0x7fe0c81430c0 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 10 monclient(hunting): dump: 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.896+0000 7fe0bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:21.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170116842 0 0) 0x7fe0ac004eb0 con 0x7fe0c80a98f0 2024-03-25T18:07:21.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0c8136c50 con 0x7fe0c80a98f0 2024-03-25T18:07:21.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe0c40019b0 con 0x7fe0c80a98f0 2024-03-25T18:07:21.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.906+0000 7fe0bdffb640 10 monclient(hunting): dump: 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:21.907 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:21.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1242324529 0 0) 0x7fe0c8136ec0 con 0x7fe0c81430c0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 >> v1:172.21.15.157:6789/0 conn(0x7fe0c813f710 legacy=0x7fe0c81dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 >> v1:172.21.15.84:6789/0 conn(0x7fe0c80a98f0 legacy=0x7fe0c81310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 1 monclient: found mon.c 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c8131980 con 0x7fe0c81430c0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 10 monclient: _finish_auth 0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0cfa33640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0cfa33640 10 monclient: _renew_subs 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0cfa33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:21.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.913+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c8135070 con 0x7fe0c81430c0 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.914+0000 7fe0cfa33640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.914+0000 7fe0cfa33640 10 monclient: _renew_subs 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.914+0000 7fe0cfa33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.914+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c80b7870 con 0x7fe0c81430c0 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.914+0000 7fe0cfa33640 1 librados: init done 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.915+0000 7fe0cfa33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.915+0000 7fe0cfa33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.915+0000 7fe0cfa33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:21.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:21.915+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c8135c10 con 0x7fe0c81430c0 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c0002080 con 0x7fe0c81430c0 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe0c0003dc0 con 0x7fe0c81430c0 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bdffb640 10 monclient: dump: 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:22.204 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.203+0000 7fe0bf7fe640 4 set_mon_vals no callback set 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.204+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fe0c0019d90 con 0x7fe0c81430c0 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.205+0000 7fe0cd7cf640 10 monclient: get_auth_request con 0x7fe0ac06df20 auth_method 0 2024-03-25T18:07:22.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.205+0000 7fe0bdffb640 10 client.4322.objecter ms_handle_connect 0x7fe0ac06df20 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.210+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fe0c0053930 con 0x7fe0c81430c0 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.210+0000 7fe0bdffb640 10 client.4322.objecter ms_dispatch 0x7fe0c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.210+0000 7fe0bdffb640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.210+0000 7fe0bdffb640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.210+0000 7fe0bdffb640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.211+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0c004f440 con 0x7fe0c81430c0 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.211+0000 7fe0bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:22.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.211+0000 7fe0bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:22.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.451+0000 7fe0cfa33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:22.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.451+0000 7fe0cfa33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:22.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.451+0000 7fe0cfa33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:07:22.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.451+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0c80627b0 con 0x7fe0c81430c0 2024-03-25T18:07:22.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.470+0000 7fe0bdffb640 1 -- 172.21.15.84:0/1148390145 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fe0c0051370 con 0x7fe0c81430c0 2024-03-25T18:07:22.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.470+0000 7fe0bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:22.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.470+0000 7fe0bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:22.470 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:22.470 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 10 librados: watch_flush enter 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 10 librados: watch_flush exit 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 10 client.4322.objecter successfully canceled tick 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 >> v1:172.21.15.157:6800/1801259026 conn(0x7fe0ac06df20 legacy=0x7fe0ac0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 10 monclient: shutdown 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 >> v1:172.21.15.84:6790/0 conn(0x7fe0c81430c0 legacy=0x7fe0c81db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 shutdown_connections 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 >> 172.21.15.84:0/1148390145 conn(0x7fe0c80b7ea0 msgr2=0x7fe0c813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.472+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 shutdown_connections 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.473+0000 7fe0cfa33640 1 -- 172.21.15.84:0/1148390145 wait complete. 2024-03-25T18:07:22.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:22.473+0000 7fe0cfa33640 1 librados: shutdown 2024-03-25T18:07:22.484 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:29.290 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:29.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.397+0000 7fe65c61e640 10 monclient: get_monmap_and_config 2024-03-25T18:07:29.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.397+0000 7fe65c61e640 10 monclient: build_initial_monmap 2024-03-25T18:07:29.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.397+0000 7fe65c61e640 10 monclient: monmap: 2024-03-25T18:07:29.398 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:29.398894+0000 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:29.398894+0000 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:29.399 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.398+0000 7fe65c61e640 1 Processor -- start 2024-03-25T18:07:29.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.398+0000 7fe65c61e640 1 -- start start 2024-03-25T18:07:29.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.398+0000 7fe65c61e640 10 monclient: init 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-b con 0x7fe65406f360 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-a con 0x7fe65406f740 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-c con 0x7fe654142e30 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe654071d20 con 0x7fe65406f740 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe654071340 con 0x7fe654142e30 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe654071620 con 0x7fe65406f360 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.399+0000 7fe65c61e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:29.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.400+0000 7fe65c61e640 10 monclient(hunting): authenticate will time out at 7093.698901s 2024-03-25T18:07:29.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.401+0000 7fe65a3ba640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe65406f360 0x7fe654142590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41936/0 (socket says 172.21.15.84:41936) 2024-03-25T18:07:29.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.401+0000 7fe65a3ba640 1 -- 172.21.15.84:0/2083691121 learned_addr learned my addr 172.21.15.84:0/2083691121 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:29.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761369701 0 0) 0x7fe654071620 con 0x7fe65406f360 2024-03-25T18:07:29.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:07:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.410+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640003520 con 0x7fe65406f360 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988965463 0 0) 0x7fe654071340 con 0x7fe654142e30 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:07:29.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.421+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654071620 con 0x7fe654142e30 2024-03-25T18:07:29.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877816968 0 0) 0x7fe654071d20 con 0x7fe65406f740 2024-03-25T18:07:29.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:07:29.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.423+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654071340 con 0x7fe65406f740 2024-03-25T18:07:29.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288696670 0 0) 0x7fe640003520 con 0x7fe65406f360 2024-03-25T18:07:29.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 >> v1:172.21.15.157:6789/0 conn(0x7fe654142e30 legacy=0x7fe65414a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 >> v1:172.21.15.84:6789/0 conn(0x7fe65406f740 legacy=0x7fe654072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: _finish_hunting 0 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 monclient: found mon.noname-b 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654071900 con 0x7fe65406f360 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: _finish_auth 0 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: _check_auth_tickets 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe650001690 con 0x7fe65406f360 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe6593b8640 10 monclient: dump: 2024-03-25T18:07:29.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe65c61e640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:07:29.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.434+0000 7fe65c61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:29.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6500017f0 con 0x7fe65406f360 2024-03-25T18:07:29.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:29.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 1 -- 172.21.15.84:0/2083691121 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe650001b80 con 0x7fe65406f360 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe65c61e640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe65c61e640 4 set_mon_vals no callback set 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.449+0000 7fe6593b8640 10 monclient: dump: 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.451 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 10 monclient: shutdown 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 1 -- 172.21.15.84:0/2083691121 >> v1:172.21.15.84:6790/0 conn(0x7fe65406f360 legacy=0x7fe654142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 1 -- 172.21.15.84:0/2083691121 shutdown_connections 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 1 -- 172.21.15.84:0/2083691121 >> 172.21.15.84:0/2083691121 conn(0x7fe6540b4b80 msgr2=0x7fe6540b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 1 -- 172.21.15.84:0/2083691121 shutdown_connections 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.450+0000 7fe65c61e640 1 -- 172.21.15.84:0/2083691121 wait complete. 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.451+0000 7fe65c61e640 10 monclient: build_initial_monmap 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.451+0000 7fe65c61e640 10 monclient: monmap: 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:29.453 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 librados: starting msgr at 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 librados: starting objecter 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 Processor -- start 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 -- start start 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 librados: setting wanted keys 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 librados: calling monclient init 2024-03-25T18:07:29.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient: init 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-a con 0x7fe65406f360 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-b con 0x7fe65406f740 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient(hunting): picked mon.noname-c con 0x7fe654142e30 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:29.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6541d33a0 con 0x7fe65406f360 2024-03-25T18:07:29.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6541d3070 con 0x7fe65406f740 2024-03-25T18:07:29.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65413f060 con 0x7fe654142e30 2024-03-25T18:07:29.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:29.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.452+0000 7fe65c61e640 10 monclient(hunting): authenticate will time out at 7093.751762s 2024-03-25T18:07:29.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.457+0000 7fe65a3ba640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe65406f360 0x7fe65413d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39222/0 (socket says 172.21.15.84:39222) 2024-03-25T18:07:29.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.457+0000 7fe65a3ba640 1 -- 172.21.15.84:0/3374381859 learned_addr learned my addr 172.21.15.84:0/3374381859 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:29.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.459+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe65406f360 2024-03-25T18:07:29.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.459+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:29.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.463+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe65406f740 2024-03-25T18:07:29.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.463+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:29.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.465+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe654142e30 2024-03-25T18:07:29.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.465+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:29.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336200195 0 0) 0x7fe6541d33a0 con 0x7fe65406f360 2024-03-25T18:07:29.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:07:29.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.466+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe638004eb0 con 0x7fe65406f360 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231784392 0 0) 0x7fe65413f060 con 0x7fe654142e30 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:07:29.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.477+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6541d33a0 con 0x7fe654142e30 2024-03-25T18:07:29.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988951720 0 0) 0x7fe6541d3070 con 0x7fe65406f740 2024-03-25T18:07:29.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:29.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:29.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:29.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:07:29.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.480+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65413f060 con 0x7fe65406f740 2024-03-25T18:07:29.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.493+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248760768 0 0) 0x7fe638004eb0 con 0x7fe65406f360 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6541d3070 con 0x7fe65406f360 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe650001b00 con 0x7fe65406f360 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 10 monclient(hunting): dump: 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.495 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.496 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.494+0000 7fe64affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:29.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686041592 0 0) 0x7fe6541d33a0 con 0x7fe654142e30 2024-03-25T18:07:29.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe638004eb0 con 0x7fe654142e30 2024-03-25T18:07:29.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe64c001b50 con 0x7fe654142e30 2024-03-25T18:07:29.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): dump: 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84661362 0 0) 0x7fe65413f060 con 0x7fe65406f740 2024-03-25T18:07:29.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6541d33a0 con 0x7fe65406f740 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe644001830 con 0x7fe65406f740 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.496+0000 7fe64affd640 10 monclient(hunting): dump: 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.499 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 431291616 0 0) 0x7fe6541d3070 con 0x7fe65406f360 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 >> v1:172.21.15.84:6790/0 conn(0x7fe654142e30 legacy=0x7fe6541ddb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 >> v1:172.21.15.157:6789/0 conn(0x7fe65406f740 legacy=0x7fe6541d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 1 monclient: found mon.a 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6541d3f40 con 0x7fe65406f360 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 10 monclient: _finish_auth 0 2024-03-25T18:07:29.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 10 monclient: _renew_subs 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6540adff0 con 0x7fe65406f360 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 10 monclient: _renew_subs 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540ab960 con 0x7fe65406f360 2024-03-25T18:07:29.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.501+0000 7fe65c61e640 1 librados: init done 2024-03-25T18:07:29.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.502+0000 7fe65c61e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:29.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.502+0000 7fe65c61e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:29.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.502+0000 7fe65c61e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:29.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.502+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6540ae2f0 con 0x7fe65406f360 2024-03-25T18:07:29.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe650004a90 con 0x7fe65406f360 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe650004da0 con 0x7fe65406f360 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe64affd640 10 monclient: dump: 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:29.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.810+0000 7fe658bb7640 4 set_mon_vals no callback set 2024-03-25T18:07:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.811+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fe65001ad70 con 0x7fe65406f360 2024-03-25T18:07:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.811+0000 7fe659bb9640 10 monclient: get_auth_request con 0x7fe63806e0f0 auth_method 0 2024-03-25T18:07:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.812+0000 7fe64affd640 10 client.4326.objecter ms_handle_connect 0x7fe63806e0f0 2024-03-25T18:07:29.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fe650054940 con 0x7fe65406f360 2024-03-25T18:07:29.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 10 client.4326.objecter ms_dispatch 0x7fe654000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:29.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:29.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:29.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-25T18:07:29.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe650050450 con 0x7fe65406f360 2024-03-25T18:07:29.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:29.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:29.818+0000 7fe64affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.068+0000 7fe65c61e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.068+0000 7fe65c61e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.068+0000 7fe65c61e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.068+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6540627b0 con 0x7fe65406f360 2024-03-25T18:07:30.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.096+0000 7fe64affd640 1 -- 172.21.15.84:0/3374381859 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fe650052380 con 0x7fe65406f360 2024-03-25T18:07:30.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.096+0000 7fe64affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:30.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.096+0000 7fe64affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:30.098 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:30.098 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 10 librados: watch_flush enter 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 10 librados: watch_flush exit 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 10 client.4326.objecter successfully canceled tick 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 >> v1:172.21.15.157:6800/1801259026 conn(0x7fe63806e0f0 legacy=0x7fe6380705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 10 monclient: shutdown 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 >> v1:172.21.15.84:6789/0 conn(0x7fe65406f360 legacy=0x7fe65413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 shutdown_connections 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 >> 172.21.15.84:0/3374381859 conn(0x7fe6540b4b80 msgr2=0x7fe6540b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 shutdown_connections 2024-03-25T18:07:30.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 -- 172.21.15.84:0/3374381859 wait complete. 2024-03-25T18:07:30.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:30.098+0000 7fe65c61e640 1 librados: shutdown 2024-03-25T18:07:30.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:36.918 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:37.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.032+0000 7f6925d7b640 10 monclient: get_monmap_and_config 2024-03-25T18:07:37.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.032+0000 7f6925d7b640 10 monclient: build_initial_monmap 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.032+0000 7f6925d7b640 10 monclient: monmap: 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:37.034036+0000 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:37.034036+0000 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:37.034 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.033+0000 7f6925d7b640 1 Processor -- start 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.033+0000 7f6925d7b640 1 -- start start 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.033+0000 7f6925d7b640 10 monclient: init 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-a con 0x7f69200a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-b con 0x7f692013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-c con 0x7f69201430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6920142520 con 0x7f69200a98f0 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200b28a0 con 0x7f69201430c0 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200b2a30 con 0x7f692013f710 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:37.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.034+0000 7f6925d7b640 10 monclient(hunting): authenticate will time out at 7101.333660s 2024-03-25T18:07:37.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.045+0000 7f691ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f69201430c0 0x7f692014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50846/0 (socket says 172.21.15.84:50846) 2024-03-25T18:07:37.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.045+0000 7f691ffff640 1 -- 172.21.15.84:0/3397059636 learned_addr learned my addr 172.21.15.84:0/3397059636 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629216473 0 0) 0x7f69200b28a0 con 0x7f69201430c0 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:07:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.052+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6908003520 con 0x7f69201430c0 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133874994 0 0) 0x7f69200b2a30 con 0x7f692013f710 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:07:37.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.064+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200b28a0 con 0x7f692013f710 2024-03-25T18:07:37.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821499583 0 0) 0x7f6920142520 con 0x7f69200a98f0 2024-03-25T18:07:37.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:07:37.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.066+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200b2a30 con 0x7f69200a98f0 2024-03-25T18:07:37.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1235427267 0 0) 0x7f6908003520 con 0x7f69201430c0 2024-03-25T18:07:37.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 >> v1:172.21.15.84:6790/0 conn(0x7f692013f710 legacy=0x7f69201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 >> v1:172.21.15.84:6789/0 conn(0x7f69200a98f0 legacy=0x7f69200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69200b44c0 con 0x7f69201430c0 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6918001a10 con 0x7f69201430c0 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f691e7fc640 10 monclient: dump: 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:37.079 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f6925d7b640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:07:37.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.077+0000 7f6925d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6918001b70 con 0x7f69201430c0 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 1 -- 172.21.15.84:0/3397059636 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6918001f80 con 0x7f69201430c0 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f691e7fc640 10 monclient: dump: 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.092 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 4 set_mon_vals no callback set 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 10 monclient: shutdown 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 1 -- 172.21.15.84:0/3397059636 >> v1:172.21.15.157:6789/0 conn(0x7f69201430c0 legacy=0x7f692014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 1 -- 172.21.15.84:0/3397059636 shutdown_connections 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.091+0000 7f6925d7b640 1 -- 172.21.15.84:0/3397059636 >> 172.21.15.84:0/3397059636 conn(0x7f69200b7ea0 msgr2=0x7f69200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.092+0000 7f6925d7b640 1 -- 172.21.15.84:0/3397059636 shutdown_connections 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.092+0000 7f6925d7b640 1 -- 172.21.15.84:0/3397059636 wait complete. 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.092+0000 7f6925d7b640 10 monclient: build_initial_monmap 2024-03-25T18:07:37.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.092+0000 7f6925d7b640 10 monclient: monmap: 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 librados: starting msgr at 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 librados: starting objecter 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 Processor -- start 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 -- start start 2024-03-25T18:07:37.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 librados: setting wanted keys 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 1 librados: calling monclient init 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 10 monclient: init 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.093+0000 7f6925d7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-a con 0x7f69200a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-b con 0x7f692013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient(hunting): picked mon.noname-c con 0x7f69201430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69201daca0 con 0x7f69200a98f0 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6920138ab0 con 0x7f692013f710 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200aba90 con 0x7f69201430c0 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:37.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.094+0000 7f6925d7b640 10 monclient(hunting): authenticate will time out at 7101.392978s 2024-03-25T18:07:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.101+0000 7f691effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f692013f710 0x7f69201db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50894/0 (socket says 172.21.15.84:50894) 2024-03-25T18:07:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.101+0000 7f691effd640 1 -- 172.21.15.84:0/1044289183 learned_addr learned my addr 172.21.15.84:0/1044289183 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:37.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.103+0000 7f6907fff640 10 client.?.objecter ms_handle_connect 0x7f692013f710 2024-03-25T18:07:37.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.103+0000 7f6907fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.104+0000 7f6907fff640 10 client.?.objecter ms_handle_connect 0x7f69201430c0 2024-03-25T18:07:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.104+0000 7f6907fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:37.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.108+0000 7f6907fff640 10 client.?.objecter ms_handle_connect 0x7f69200a98f0 2024-03-25T18:07:37.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.108+0000 7f6907fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482752133 0 0) 0x7f69200aba90 con 0x7f69201430c0 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:07:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.113+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c004eb0 con 0x7f69201430c0 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075126776 0 0) 0x7f6920138ab0 con 0x7f692013f710 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:07:37.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.118+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200aba90 con 0x7f692013f710 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057199938 0 0) 0x7f69201daca0 con 0x7f69200a98f0 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:07:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.129+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920138ab0 con 0x7f69200a98f0 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108719278 0 0) 0x7f69200aba90 con 0x7f692013f710 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69201daca0 con 0x7f692013f710 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f69140019b0 con 0x7f692013f710 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 10 monclient(hunting): dump: 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.130+0000 7f6907fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700618361 0 0) 0x7f6920138ab0 con 0x7f69200a98f0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69200aba90 con 0x7f69200a98f0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637789416 0 0) 0x7f69201daca0 con 0x7f692013f710 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 >> v1:172.21.15.84:6790/0 conn(0x7f69201430c0 legacy=0x7f69201d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 >> v1:172.21.15.84:6789/0 conn(0x7f69200a98f0 legacy=0x7f69201383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 monclient: found mon.b 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69201d41e0 con 0x7f692013f710 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 10 monclient: _finish_auth 0 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:37.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.140+0000 7f6907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _renew_subs 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69201db840 con 0x7f692013f710 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _renew_subs 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920138f00 con 0x7f692013f710 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 1 librados: init done 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.141+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69201dae50 con 0x7f692013f710 2024-03-25T18:07:37.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6914003610 con 0x7f692013f710 2024-03-25T18:07:37.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:37.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6914003800 con 0x7f692013f710 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f6907fff640 10 monclient: dump: 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:37.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.439+0000 7f691dffb640 4 set_mon_vals no callback set 2024-03-25T18:07:37.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.440+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f69140039e0 con 0x7f692013f710 2024-03-25T18:07:37.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.441+0000 7f691f7fe640 10 monclient: get_auth_request con 0x7f690c06e080 auth_method 0 2024-03-25T18:07:37.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.441+0000 7f6907fff640 10 client.4333.objecter ms_handle_connect 0x7f690c06e080 2024-03-25T18:07:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f6914054730 con 0x7f692013f710 2024-03-25T18:07:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 10 client.4333.objecter ms_dispatch 0x7f6920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6914050070 con 0x7f692013f710 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.445+0000 7f6907fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:37.601 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:34.420 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.649 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:34.468 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.677 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:34.496 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.676+0000 7f6925d7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.676+0000 7f6925d7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.676+0000 7f6925d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:37.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.676+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69200627b0 con 0x7f692013f710 2024-03-25T18:07:37.730 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:34.550 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.736 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:34.520 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.741 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:34.525 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.743 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:34.562 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.758+0000 7f6907fff640 1 -- 172.21.15.84:0/1044289183 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f6914052170 con 0x7f692013f710 2024-03-25T18:07:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.758+0000 7f6907fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.758+0000 7f6907fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:37.760 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:37.760 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:37.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 10 librados: watch_flush enter 2024-03-25T18:07:37.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 10 librados: watch_flush exit 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 10 client.4333.objecter successfully canceled tick 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 >> v1:172.21.15.157:6800/1801259026 conn(0x7f690c06e080 legacy=0x7f690c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 10 monclient: shutdown 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 >> v1:172.21.15.157:6789/0 conn(0x7f692013f710 legacy=0x7f69201db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 shutdown_connections 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 >> 172.21.15.84:0/1044289183 conn(0x7f69200b7ea0 msgr2=0x7f6920074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 shutdown_connections 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 -- 172.21.15.84:0/1044289183 wait complete. 2024-03-25T18:07:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:37.761+0000 7f6925d7b640 1 librados: shutdown 2024-03-25T18:07:37.765 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:34.549 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.770 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:34.589 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:37.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:44.580 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:44.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.692+0000 7fd4268eb640 10 monclient: get_monmap_and_config 2024-03-25T18:07:44.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.692+0000 7fd4268eb640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.692+0000 7fd4268eb640 10 monclient: monmap: 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:44.693530+0000 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:44.693530+0000 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:44.736 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.693+0000 7fd4268eb640 1 Processor -- start 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.693+0000 7fd4268eb640 1 -- start start 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.693+0000 7fd4268eb640 10 monclient: init 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd4200a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-b con 0x7fd42013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-a con 0x7fd4201430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420142520 con 0x7fd4201430c0 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200b28a0 con 0x7fd4200a98f0 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200b2a30 con 0x7fd42013f710 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.694+0000 7fd4268eb640 10 monclient(hunting): authenticate will time out at 7108.993282s 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.706+0000 7fd41ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd4200a98f0 0x7fd4200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:49292/0 (socket says 172.21.15.84:49292) 2024-03-25T18:07:44.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.706+0000 7fd41ffff640 1 -- 172.21.15.84:0/585011011 learned_addr learned my addr 172.21.15.84:0/585011011 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139190588 0 0) 0x7fd4200b2a30 con 0x7fd42013f710 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.714+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c003520 con 0x7fd42013f710 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302299270 0 0) 0x7fd4200b28a0 con 0x7fd4200a98f0 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.720+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4200b2a30 con 0x7fd4200a98f0 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2416063133 0 0) 0x7fd40c003520 con 0x7fd42013f710 2024-03-25T18:07:44.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 >> v1:172.21.15.157:6789/0 conn(0x7fd4200a98f0 legacy=0x7fd4200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 >> v1:172.21.15.84:6789/0 conn(0x7fd4201430c0 legacy=0x7fd42014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 monclient: found mon.noname-b 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4200b44c0 con 0x7fd42013f710 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd4268eb640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd4268eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd408001690 con 0x7fd42013f710 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:44.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd41effd640 10 monclient: dump: 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.728+0000 7fd4268eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4080017f0 con 0x7fd42013f710 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 1 -- 172.21.15.84:0/585011011 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd408001c00 con 0x7fd42013f710 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd4268eb640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd4268eb640 4 set_mon_vals no callback set 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.735+0000 7fd41effd640 10 monclient: dump: 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:44.745 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 10 monclient: shutdown 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 1 -- 172.21.15.84:0/585011011 >> v1:172.21.15.84:6790/0 conn(0x7fd42013f710 legacy=0x7fd4201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 1 -- 172.21.15.84:0/585011011 shutdown_connections 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 1 -- 172.21.15.84:0/585011011 >> 172.21.15.84:0/585011011 conn(0x7fd4200b7ea0 msgr2=0x7fd4200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 1 -- 172.21.15.84:0/585011011 shutdown_connections 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.736+0000 7fd4268eb640 1 -- 172.21.15.84:0/585011011 wait complete. 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 10 monclient: monmap: 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.746 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 librados: starting msgr at 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 librados: starting objecter 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 Processor -- start 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 -- start start 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 librados: setting wanted keys 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 1 librados: calling monclient init 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.737+0000 7fd4268eb640 10 monclient: init 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-b con 0x7fd4200a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd42013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:44.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient(hunting): picked mon.noname-a con 0x7fd4201430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:44.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42013be70 con 0x7fd4201430c0 2024-03-25T18:07:44.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420131660 con 0x7fd4200a98f0 2024-03-25T18:07:44.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4201317f0 con 0x7fd42013f710 2024-03-25T18:07:44.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.738+0000 7fd4268eb640 10 monclient(hunting): authenticate will time out at 7109.037461s 2024-03-25T18:07:44.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.748+0000 7fd424e88640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fd4201430c0 0x7fd4201e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55572/0 (socket says 172.21.15.84:55572) 2024-03-25T18:07:44.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.748+0000 7fd424e88640 1 -- 172.21.15.84:0/4134993547 learned_addr learned my addr 172.21.15.84:0/4134993547 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:44.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.754+0000 7fd41cff9640 10 client.?.objecter ms_handle_connect 0x7fd4201430c0 2024-03-25T18:07:44.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.755+0000 7fd41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.755+0000 7fd41cff9640 10 client.?.objecter ms_handle_connect 0x7fd4200a98f0 2024-03-25T18:07:44.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.755+0000 7fd41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.761+0000 7fd41cff9640 10 client.?.objecter ms_handle_connect 0x7fd42013f710 2024-03-25T18:07:44.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.761+0000 7fd41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018616529 0 0) 0x7fd42013be70 con 0x7fd4201430c0 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:07:44.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.764+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd400004eb0 con 0x7fd4201430c0 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693751658 0 0) 0x7fd420131660 con 0x7fd4200a98f0 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:07:44.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.772+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42013be70 con 0x7fd4200a98f0 2024-03-25T18:07:44.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908774736 0 0) 0x7fd4201317f0 con 0x7fd42013f710 2024-03-25T18:07:44.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:07:44.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.782+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd420131660 con 0x7fd42013f710 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2078271494 0 0) 0x7fd400004eb0 con 0x7fd4201430c0 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4201317f0 con 0x7fd4201430c0 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd418001c30 con 0x7fd4201430c0 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.791+0000 7fd41cff9640 10 monclient(hunting): dump: 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:44.793 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:44.794 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:44.794 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.792+0000 7fd41cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619052102 0 0) 0x7fd42013be70 con 0x7fd4200a98f0 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd400004eb0 con 0x7fd4200a98f0 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd410001930 con 0x7fd4200a98f0 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.797+0000 7fd41cff9640 10 monclient(hunting): dump: 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.798 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:44.799 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:44.799 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:44.799 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123488341 0 0) 0x7fd420131660 con 0x7fd42013f710 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd400005ed0 con 0x7fd42013f710 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd408001b00 con 0x7fd42013f710 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.806+0000 7fd41cff9640 10 monclient(hunting): dump: 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:44.808 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:44.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 457521227 0 0) 0x7fd4201317f0 con 0x7fd4201430c0 2024-03-25T18:07:44.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 >> v1:172.21.15.84:6790/0 conn(0x7fd42013f710 legacy=0x7fd4201d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 >> v1:172.21.15.157:6789/0 conn(0x7fd4200a98f0 legacy=0x7fd420130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 1 monclient: found mon.a 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420131980 con 0x7fd4201430c0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd41cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 10 monclient: _renew_subs 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd420135c60 con 0x7fd4201430c0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 10 monclient: _renew_subs 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4200b7770 con 0x7fd4201430c0 2024-03-25T18:07:44.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.811+0000 7fd4268eb640 1 librados: init done 2024-03-25T18:07:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.812+0000 7fd4268eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.812+0000 7fd4268eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.812+0000 7fd4268eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:44.812+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd420074cc0 con 0x7fd4201430c0 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd418003990 con 0x7fd4201430c0 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd418003da0 con 0x7fd4201430c0 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41cff9640 10 monclient: dump: 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:45.153 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:45.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.152+0000 7fd41e7fc640 4 set_mon_vals no callback set 2024-03-25T18:07:45.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.153+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fd418003fc0 con 0x7fd4201430c0 2024-03-25T18:07:45.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.153+0000 7fd41ffff640 10 monclient: get_auth_request con 0x7fd40006e510 auth_method 0 2024-03-25T18:07:45.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.154+0000 7fd41cff9640 10 client.4338.objecter ms_handle_connect 0x7fd40006e510 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fd418054a90 con 0x7fd4201430c0 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 10 client.4338.objecter ms_dispatch 0x7fd420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4180505a0 con 0x7fd4201430c0 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.159+0000 7fd41cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:45.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.387+0000 7fd4268eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:45.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.387+0000 7fd4268eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:45.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.387+0000 7fd4268eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:07:45.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.387+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4201371d0 con 0x7fd4201430c0 2024-03-25T18:07:45.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.432+0000 7fd41cff9640 1 -- 172.21.15.84:0/4134993547 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fd4180524d0 con 0x7fd4201430c0 2024-03-25T18:07:45.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.432+0000 7fd41cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:45.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.432+0000 7fd41cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:45.433 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:45.433 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.434+0000 7fd4268eb640 10 librados: watch_flush enter 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.434+0000 7fd4268eb640 10 librados: watch_flush exit 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.434+0000 7fd4268eb640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.434+0000 7fd4268eb640 10 client.4338.objecter successfully canceled tick 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.434+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 >> v1:172.21.15.157:6800/1801259026 conn(0x7fd40006e510 legacy=0x7fd4000709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 10 monclient: shutdown 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 >> v1:172.21.15.84:6789/0 conn(0x7fd4201430c0 legacy=0x7fd4201e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 shutdown_connections 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 >> 172.21.15.84:0/4134993547 conn(0x7fd4200b7ea0 msgr2=0x7fd420074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 shutdown_connections 2024-03-25T18:07:45.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 -- 172.21.15.84:0/4134993547 wait complete. 2024-03-25T18:07:45.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:45.435+0000 7fd4268eb640 1 librados: shutdown 2024-03-25T18:07:45.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:52.253 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 10 monclient: get_monmap_and_config 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 10 monclient: build_initial_monmap 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 10 monclient: monmap: 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:07:52.366447+0000 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:07:52.366447+0000 2024-03-25T18:07:52.366 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:52.367 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.367 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:52.367 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:07:52.367 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:07:52.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 1 Processor -- start 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 1 -- start start 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.365+0000 7f372b9c8640 10 monclient: init 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f37240a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f372413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f37241430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3724142520 con 0x7f372413f710 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37240b28a0 con 0x7f37241430c0 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37240b2a30 con 0x7f37240a98f0 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.366+0000 7f372b9c8640 10 monclient(hunting): authenticate will time out at 7116.665778s 2024-03-25T18:07:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.381+0000 7f3729764640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f37240a98f0 0x7f37240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45606/0 (socket says 172.21.15.84:45606) 2024-03-25T18:07:52.382 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.381+0000 7f3729764640 1 -- 172.21.15.84:0/1447422950 learned_addr learned my addr 172.21.15.84:0/1447422950 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7952436 0 0) 0x7f3724142520 con 0x7f372413f710 2024-03-25T18:07:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:07:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.393+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3710003520 con 0x7f372413f710 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551011974 0 0) 0x7f37240b2a30 con 0x7f37240a98f0 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:07:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.398+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724142520 con 0x7f37240a98f0 2024-03-25T18:07:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178737699 0 0) 0x7f37240b28a0 con 0x7f37241430c0 2024-03-25T18:07:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:07:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.402+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37240b2a30 con 0x7f37241430c0 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2360978143 0 0) 0x7f3710003520 con 0x7f372413f710 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 >> v1:172.21.15.84:6790/0 conn(0x7f37240a98f0 legacy=0x7f37240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 >> v1:172.21.15.157:6789/0 conn(0x7f37241430c0 legacy=0x7f372414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 1 monclient: found mon.noname-a 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.405+0000 7f371bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240b44c0 con 0x7f372413f710 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 10 monclient: _finish_auth 0 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:52.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3714001690 con 0x7f372413f710 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f371bfff640 10 monclient: dump: 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f372b9c8640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:07:52.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.406+0000 7f372b9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:52.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37140017f0 con 0x7f372413f710 2024-03-25T18:07:52.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:52.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 1 -- 172.21.15.84:0/1447422950 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3714001c00 con 0x7f372413f710 2024-03-25T18:07:52.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f372b9c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:52.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f372b9c8640 4 set_mon_vals no callback set 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.419+0000 7f371bfff640 10 monclient: dump: 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.421 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 10 monclient: shutdown 2024-03-25T18:07:52.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:52.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 1 -- 172.21.15.84:0/1447422950 >> v1:172.21.15.84:6789/0 conn(0x7f372413f710 legacy=0x7f37241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 1 -- 172.21.15.84:0/1447422950 shutdown_connections 2024-03-25T18:07:52.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 1 -- 172.21.15.84:0/1447422950 >> 172.21.15.84:0/1447422950 conn(0x7f37240b7ea0 msgr2=0x7f37240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 1 -- 172.21.15.84:0/1447422950 shutdown_connections 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.420+0000 7f372b9c8640 1 -- 172.21.15.84:0/1447422950 wait complete. 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 10 monclient: build_initial_monmap 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 10 monclient: monmap: 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 librados: starting msgr at 2024-03-25T18:07:52.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 librados: starting objecter 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 Processor -- start 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 -- start start 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 librados: setting wanted keys 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 1 librados: calling monclient init 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.421+0000 7f372b9c8640 10 monclient: init 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f37240a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f372413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f37241430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3724139520 con 0x7f37240a98f0 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3724138b40 con 0x7f372413f710 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3724138e20 con 0x7f37241430c0 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:52.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.422+0000 7f372b9c8640 10 monclient(hunting): authenticate will time out at 7116.721587s 2024-03-25T18:07:52.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.437+0000 7f3728f63640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f372413f710 0x7f3724137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:37192/0 (socket says 172.21.15.84:37192) 2024-03-25T18:07:52.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.437+0000 7f3728f63640 1 -- 172.21.15.84:0/1529745338 learned_addr learned my addr 172.21.15.84:0/1529745338 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:07:52.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.443+0000 7f3719ffb640 10 client.?.objecter ms_handle_connect 0x7f37240a98f0 2024-03-25T18:07:52.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.443+0000 7f3719ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:52.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.444+0000 7f3719ffb640 10 client.?.objecter ms_handle_connect 0x7f372413f710 2024-03-25T18:07:52.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.444+0000 7f3719ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:52.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.445+0000 7f3719ffb640 10 client.?.objecter ms_handle_connect 0x7f37241430c0 2024-03-25T18:07:52.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.445+0000 7f3719ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991847070 0 0) 0x7f3724138e20 con 0x7f37241430c0 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:07:52.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.462+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3708004eb0 con 0x7f37241430c0 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516136590 0 0) 0x7f3724138b40 con 0x7f372413f710 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:07:52.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.465+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724138e20 con 0x7f372413f710 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258706169 0 0) 0x7f3724139520 con 0x7f37240a98f0 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:07:52.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.471+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724138b40 con 0x7f37240a98f0 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.485+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941317617 0 0) 0x7f3724138e20 con 0x7f372413f710 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3724139520 con 0x7f372413f710 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3714001b40 con 0x7f372413f710 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 10 monclient(hunting): dump: 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.487 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.488 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.486+0000 7f3719ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834896236 0 0) 0x7f3708004eb0 con 0x7f37241430c0 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3724138e20 con 0x7f37241430c0 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f371c001bd0 con 0x7f37241430c0 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 10 monclient(hunting): dump: 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.498 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.496+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3539382609 0 0) 0x7f3724138b40 con 0x7f37240a98f0 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.497+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3708004eb0 con 0x7f37240a98f0 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.497+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f37200018f0 con 0x7f37240a98f0 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.497+0000 7f3719ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.497+0000 7f3719ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.497+0000 7f3719ffb640 10 monclient(hunting): dump: 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.499 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1753947051 0 0) 0x7f3724139520 con 0x7f372413f710 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 >> v1:172.21.15.84:6790/0 conn(0x7f37241430c0 legacy=0x7f37241e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 >> v1:172.21.15.84:6789/0 conn(0x7f37240a98f0 legacy=0x7f3724137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 1 monclient: found mon.b 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724139100 con 0x7f372413f710 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 10 monclient: _finish_auth 0 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f3719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:07:52.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 10 monclient: _renew_subs 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3724136aa0 con 0x7f372413f710 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 10 monclient: _renew_subs 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f372413ba10 con 0x7f372413f710 2024-03-25T18:07:52.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.521+0000 7f372b9c8640 1 librados: init done 2024-03-25T18:07:52.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.522+0000 7f372b9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:52.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.522+0000 7f372b9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:52.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.522+0000 7f372b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:52.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.522+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37240627b0 con 0x7f372413f710 2024-03-25T18:07:52.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37140038a0 con 0x7f372413f710 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3714001250 con 0x7f372413f710 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f3719ffb640 10 monclient: dump: 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:07:52.816 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.814+0000 7f371b7fe640 4 set_mon_vals no callback set 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.815+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f3714003cf0 con 0x7f372413f710 2024-03-25T18:07:52.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.816+0000 7f3729764640 10 monclient: get_auth_request con 0x7f370806e0f0 auth_method 0 2024-03-25T18:07:52.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.816+0000 7f3719ffb640 10 client.4345.objecter ms_handle_connect 0x7f370806e0f0 2024-03-25T18:07:52.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f37140547e0 con 0x7f372413f710 2024-03-25T18:07:52.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 10 client.4345.objecter ms_dispatch 0x7f3724000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:52.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:52.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:52.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T18:07:52.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37140502f0 con 0x7f372413f710 2024-03-25T18:07:52.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:52.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:52.821+0000 7f3719ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:53.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.030+0000 7f372b9c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:53.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.030+0000 7f372b9c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:53.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.030+0000 7f372b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:07:53.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.030+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f372413df90 con 0x7f372413f710 2024-03-25T18:07:53.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.115+0000 7f3719ffb640 1 -- 172.21.15.84:0/1529745338 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f3714052220 con 0x7f372413f710 2024-03-25T18:07:53.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.115+0000 7f3719ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:53.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.115+0000 7f3719ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:53.116 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:07:53.116 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 10 librados: watch_flush enter 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 10 librados: watch_flush exit 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 10 client.4345.objecter successfully canceled tick 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 >> v1:172.21.15.157:6800/1801259026 conn(0x7f370806e0f0 legacy=0x7f37080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 10 monclient: shutdown 2024-03-25T18:07:53.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 >> v1:172.21.15.157:6789/0 conn(0x7f372413f710 legacy=0x7f3724137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 shutdown_connections 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.117+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 >> 172.21.15.84:0/1529745338 conn(0x7f37240b7ea0 msgr2=0x7f3724074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.118+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 shutdown_connections 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.118+0000 7f372b9c8640 1 -- 172.21.15.84:0/1529745338 wait complete. 2024-03-25T18:07:53.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:07:53.118+0000 7f372b9c8640 1 librados: shutdown 2024-03-25T18:07:53.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:57.656 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:54.475 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.710 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:54.529 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.712 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:54.532 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.742 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:54.561 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.766 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:54.550 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.768 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:54.588 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.773 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:54.556 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.798 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:54.617 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:57.800 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:02:54.584 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:59.938 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.050+0000 7feef7d58640 10 monclient: get_monmap_and_config 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.050+0000 7feef7d58640 10 monclient: build_initial_monmap 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.050+0000 7feef7d58640 10 monclient: monmap: 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:00.051 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:00.051784+0000 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:00.051784+0000 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:00.052 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.050+0000 7feef7d58640 1 Processor -- start 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 1 -- start start 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient: init 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-b con 0x7feef00b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-c con 0x7feef00b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-a con 0x7feef014be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00af940 con 0x7feef014be90 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00b0a70 con 0x7feef00b1730 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00b0d50 con 0x7feef00b4d60 2024-03-25T18:08:00.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:00.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.051+0000 7feef7d58640 10 monclient(hunting): authenticate will time out at 7124.350745s 2024-03-25T18:08:00.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.054+0000 7feef5af4640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7feef00b4d60 0x7feef013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52320/0 (socket says 172.21.15.84:52320) 2024-03-25T18:08:00.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.054+0000 7feef5af4640 1 -- 172.21.15.84:0/694751412 learned_addr learned my addr 172.21.15.84:0/694751412 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159603974 0 0) 0x7feef00b0d50 con 0x7feef00b4d60 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:08:00.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.067+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feedc003520 con 0x7feef00b4d60 2024-03-25T18:08:00.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520253409 0 0) 0x7feef00b0a70 con 0x7feef00b1730 2024-03-25T18:08:00.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:08:00.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.073+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef00b0d50 con 0x7feef00b1730 2024-03-25T18:08:00.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789440041 0 0) 0x7feef00af940 con 0x7feef014be90 2024-03-25T18:08:00.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:08:00.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.075+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef00b0a70 con 0x7feef014be90 2024-03-25T18:08:00.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649595951 0 0) 0x7feef00b0d50 con 0x7feef00b1730 2024-03-25T18:08:00.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 >> v1:172.21.15.84:6790/0 conn(0x7feef00b4d60 legacy=0x7feef013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 >> v1:172.21.15.84:6789/0 conn(0x7feef014be90 legacy=0x7feef014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: _finish_hunting 0 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 monclient: found mon.noname-c 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00b1030 con 0x7feef00b1730 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: _finish_auth 0 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: _check_auth_tickets 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7feee0001670 con 0x7feef00b1730 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef4af2640 10 monclient: dump: 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.081 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef7d58640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:08:00.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.079+0000 7feef7d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:00.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.086+0000 7feef4af2640 1 -- 172.21.15.84:0/694751412 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feee00017d0 con 0x7feef00b1730 2024-03-25T18:08:00.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.086+0000 7feef4af2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:00.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.086+0000 7feef7d58640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:00.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.086+0000 7feef7d58640 4 set_mon_vals no callback set 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 10 monclient: shutdown 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 1 -- 172.21.15.84:0/694751412 >> v1:172.21.15.157:6789/0 conn(0x7feef00b1730 legacy=0x7feef00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 1 -- 172.21.15.84:0/694751412 shutdown_connections 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 1 -- 172.21.15.84:0/694751412 >> 172.21.15.84:0/694751412 conn(0x7feef00617a0 msgr2=0x7feef0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 1 -- 172.21.15.84:0/694751412 shutdown_connections 2024-03-25T18:08:00.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.087+0000 7feef7d58640 1 -- 172.21.15.84:0/694751412 wait complete. 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.088+0000 7feef7d58640 10 monclient: build_initial_monmap 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.088+0000 7feef7d58640 10 monclient: monmap: 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:00.089 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:00.090 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:00.090 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:00.090 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:00.090 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 librados: starting msgr at 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 librados: starting objecter 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 Processor -- start 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 -- start start 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 librados: setting wanted keys 2024-03-25T18:08:00.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 librados: calling monclient init 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient: init 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-a con 0x7feef00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-c con 0x7feef00b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient(hunting): picked mon.noname-b con 0x7feef014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef0140a10 con 0x7feef00b1730 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef0132520 con 0x7feef014be90 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef01326b0 con 0x7feef00b4d60 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.089+0000 7feef7d58640 10 monclient(hunting): authenticate will time out at 7124.388860s 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.090+0000 7feef52f3640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7feef00b4d60 0x7feef01dad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52364/0 (socket says 172.21.15.84:52364) 2024-03-25T18:08:00.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.090+0000 7feef52f3640 1 -- 172.21.15.84:0/254714727 learned_addr learned my addr 172.21.15.84:0/254714727 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:00.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.091+0000 7feee67fc640 10 client.?.objecter ms_handle_connect 0x7feef00b4d60 2024-03-25T18:08:00.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.091+0000 7feee67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:00.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043339225 0 0) 0x7feef01326b0 con 0x7feef00b4d60 2024-03-25T18:08:00.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:08:00.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.102+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4004eb0 con 0x7feef00b4d60 2024-03-25T18:08:00.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.106+0000 7feee67fc640 10 client.?.objecter ms_handle_connect 0x7feef00b1730 2024-03-25T18:08:00.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.106+0000 7feee67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.109+0000 7feee67fc640 10 client.?.objecter ms_handle_connect 0x7feef014be90 2024-03-25T18:08:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.109+0000 7feee67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252320818 0 0) 0x7feef0140a10 con 0x7feef00b1730 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:08:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.117+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef01326b0 con 0x7feef00b1730 2024-03-25T18:08:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1335008104 0 0) 0x7feed4004eb0 con 0x7feef00b4d60 2024-03-25T18:08:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feef0140a10 con 0x7feef00b4d60 2024-03-25T18:08:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7feee0001d80 con 0x7feef00b4d60 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 10 monclient(hunting): dump: 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:00.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.122+0000 7feee67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438722156 0 0) 0x7feef0132520 con 0x7feef014be90 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:08:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.124+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4004eb0 con 0x7feef014be90 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3805079921 0 0) 0x7feef0140a10 con 0x7feef00b4d60 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 >> v1:172.21.15.157:6789/0 conn(0x7feef014be90 legacy=0x7feef01db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 >> v1:172.21.15.84:6789/0 conn(0x7feef00b1730 legacy=0x7feef0131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 1 monclient: found mon.c 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0132840 con 0x7feef00b4d60 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:00.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feef7d58640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feef7d58640 10 monclient: _renew_subs 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feef7d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef0137720 con 0x7feef00b4d60 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.140+0000 7feef7d58640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.141+0000 7feef7d58640 10 monclient: _renew_subs 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.141+0000 7feef7d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.141+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feef0144da0 con 0x7feef00b4d60 2024-03-25T18:08:00.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.141+0000 7feef7d58640 1 librados: init done 2024-03-25T18:08:00.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.142+0000 7feef7d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:00.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.142+0000 7feef7d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:00.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.142+0000 7feef7d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:00.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.142+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00627b0 con 0x7feef00b4d60 2024-03-25T18:08:00.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.438+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feee00038a0 con 0x7feef00b4d60 2024-03-25T18:08:00.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.438+0000 7feee67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:00.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.438+0000 7feee7fff640 4 set_mon_vals no callback set 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.449+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7feee0003c30 con 0x7feef00b4d60 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.449+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.449+0000 7feee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.449+0000 7feee67fc640 10 monclient: dump: 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:00.450 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:00.451 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:00.451 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:00.451 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:00.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.450+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7feee0003e10 con 0x7feef00b4d60 2024-03-25T18:08:00.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.450+0000 7feef5af4640 10 monclient: get_auth_request con 0x7feed406de00 auth_method 0 2024-03-25T18:08:00.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.451+0000 7feee67fc640 10 client.4352.objecter ms_handle_connect 0x7feed406de00 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7feee0054a30 con 0x7feef00b4d60 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 10 client.4352.objecter ms_dispatch 0x7feef0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feee0050370 con 0x7feef00b4d60 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:00.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.456+0000 7feee67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:00.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.702+0000 7feef7d58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.702+0000 7feef7d58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.702+0000 7feef7d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:00.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.702+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feef00b28a0 con 0x7feef00b4d60 2024-03-25T18:08:00.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.733+0000 7feee67fc640 1 -- 172.21.15.84:0/254714727 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7feee0052470 con 0x7feef00b4d60 2024-03-25T18:08:00.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.733+0000 7feee67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.733+0000 7feee67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:00.734 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:00.734 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 10 librados: watch_flush enter 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 10 librados: watch_flush exit 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 10 client.4352.objecter successfully canceled tick 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 >> v1:172.21.15.157:6800/1801259026 conn(0x7feed406de00 legacy=0x7feed40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 10 monclient: shutdown 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 >> v1:172.21.15.84:6790/0 conn(0x7feef00b4d60 legacy=0x7feef01dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 shutdown_connections 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 >> 172.21.15.84:0/254714727 conn(0x7feef00617a0 msgr2=0x7feef00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 shutdown_connections 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 -- 172.21.15.84:0/254714727 wait complete. 2024-03-25T18:08:00.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:00.735+0000 7feef7d58640 1 librados: shutdown 2024-03-25T18:08:00.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:07.553 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.671+0000 7f47fa2b3640 10 monclient: get_monmap_and_config 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.671+0000 7f47fa2b3640 10 monclient: build_initial_monmap 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.671+0000 7f47fa2b3640 10 monclient: monmap: 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.671 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:07.671648+0000 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:07.671648+0000 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:07.672 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.672+0000 7f47fa2b3640 1 Processor -- start 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.672+0000 7f47fa2b3640 1 -- start start 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.672+0000 7f47fa2b3640 10 monclient: init 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f47f40b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f47f40b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f47f414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:07.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40af940 con 0x7f47f414be90 2024-03-25T18:08:07.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40b0a70 con 0x7f47f40b4d60 2024-03-25T18:08:07.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40b0d50 con 0x7f47f40b1730 2024-03-25T18:08:07.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:07.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.673+0000 7f47fa2b3640 10 monclient(hunting): authenticate will time out at 7131.971138s 2024-03-25T18:08:07.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.680+0000 7f47f3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f47f40b4d60 0x7f47f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:46008/0 (socket says 172.21.15.84:46008) 2024-03-25T18:08:07.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.680+0000 7f47f3fff640 1 -- 172.21.15.84:0/438440419 learned_addr learned my addr 172.21.15.84:0/438440419 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:07.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950320176 0 0) 0x7f47f40af940 con 0x7f47f414be90 2024-03-25T18:08:07.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.696+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47dc003520 con 0x7f47f414be90 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580115585 0 0) 0x7f47f40b0a70 con 0x7f47f40b4d60 2024-03-25T18:08:07.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:08:07.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.697+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f40af940 con 0x7f47f40b4d60 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.706+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951846610 0 0) 0x7f47dc003520 con 0x7f47f414be90 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 >> v1:172.21.15.84:6790/0 conn(0x7f47f40b1730 legacy=0x7f47f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 >> v1:172.21.15.157:6789/0 conn(0x7f47f40b4d60 legacy=0x7f47f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:08:07.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f40b1030 con 0x7f47f414be90 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47ec001a10 con 0x7f47f414be90 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47f2ffd640 10 monclient: dump: 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:07.708 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:07.709 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:07.709 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47fa2b3640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:08:07.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.707+0000 7f47fa2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47ec001b70 con 0x7f47f414be90 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 1 -- 172.21.15.84:0/438440419 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47ec001f80 con 0x7f47f414be90 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47fa2b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47fa2b3640 4 set_mon_vals no callback set 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.717+0000 7f47f2ffd640 10 monclient: dump: 2024-03-25T18:08:07.717 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:07.718 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 10 monclient: shutdown 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 1 -- 172.21.15.84:0/438440419 >> v1:172.21.15.84:6789/0 conn(0x7f47f414be90 legacy=0x7f47f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 1 -- 172.21.15.84:0/438440419 shutdown_connections 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 1 -- 172.21.15.84:0/438440419 >> 172.21.15.84:0/438440419 conn(0x7f47f40617a0 msgr2=0x7f47f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 1 -- 172.21.15.84:0/438440419 shutdown_connections 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.718+0000 7f47fa2b3640 1 -- 172.21.15.84:0/438440419 wait complete. 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 10 monclient: build_initial_monmap 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 10 monclient: monmap: 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:07.720 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 librados: starting msgr at 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 librados: starting objecter 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 Processor -- start 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 -- start start 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 librados: setting wanted keys 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 1 librados: calling monclient init 2024-03-25T18:08:07.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.719+0000 7f47fa2b3640 10 monclient: init 2024-03-25T18:08:07.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:07.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:07.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f47f40b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:07.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f47f40b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:07.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f47f414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:07.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f4137850 con 0x7f47f414be90 2024-03-25T18:08:07.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f4137b30 con 0x7f47f40b4d60 2024-03-25T18:08:07.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f4144720 con 0x7f47f40b1730 2024-03-25T18:08:07.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:07.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.720+0000 7f47fa2b3640 10 monclient(hunting): authenticate will time out at 7132.018738s 2024-03-25T18:08:07.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.725+0000 7f47f8850640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f47f414be90 0x7f47f4144010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:36870/0 (socket says 172.21.15.84:36870) 2024-03-25T18:08:07.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.725+0000 7f47f8850640 1 -- 172.21.15.84:0/513143757 learned_addr learned my addr 172.21.15.84:0/513143757 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:07.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.726+0000 7f47f0ff9640 10 client.?.objecter ms_handle_connect 0x7f47f414be90 2024-03-25T18:08:07.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.726+0000 7f47f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.738+0000 7f47f0ff9640 10 client.?.objecter ms_handle_connect 0x7f47f40b4d60 2024-03-25T18:08:07.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.738+0000 7f47f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888998093 0 0) 0x7f47f4137850 con 0x7f47f414be90 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:08:07.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.741+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e0004eb0 con 0x7f47f414be90 2024-03-25T18:08:07.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.742+0000 7f47f0ff9640 10 client.?.objecter ms_handle_connect 0x7f47f40b1730 2024-03-25T18:08:07.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.742+0000 7f47f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163431781 0 0) 0x7f47f4144720 con 0x7f47f40b1730 2024-03-25T18:08:07.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:08:07.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.753+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f4137850 con 0x7f47f40b1730 2024-03-25T18:08:07.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802295219 0 0) 0x7f47f4137b30 con 0x7f47f40b4d60 2024-03-25T18:08:07.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:08:07.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.765+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f4144720 con 0x7f47f40b4d60 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83528293 0 0) 0x7f47e0004eb0 con 0x7f47f414be90 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f4137b30 con 0x7f47f414be90 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47ec001f80 con 0x7f47f414be90 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.774 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.773+0000 7f47f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2475647290 0 0) 0x7f47f4137850 con 0x7f47f40b1730 2024-03-25T18:08:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47e0004eb0 con 0x7f47f40b1730 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47e4001930 con 0x7f47f40b1730 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.790+0000 7f47f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:07.791 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1980418942 0 0) 0x7f47f4144720 con 0x7f47f40b4d60 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f4137850 con 0x7f47f40b4d60 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47e8001860 con 0x7f47f40b4d60 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.792+0000 7f47f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:07.793 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.794 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:07.794 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:07.794 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:07.794 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:07.794 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:07.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2977584115 0 0) 0x7f47f4137b30 con 0x7f47f414be90 2024-03-25T18:08:07.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 >> v1:172.21.15.84:6790/0 conn(0x7f47f40b1730 legacy=0x7f47f4137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 >> v1:172.21.15.157:6789/0 conn(0x7f47f40b4d60 legacy=0x7f47f4143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 1 monclient: found mon.a 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4136760 con 0x7f47f414be90 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 10 monclient: _renew_subs 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f4144b90 con 0x7f47f414be90 2024-03-25T18:08:07.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 10 monclient: _renew_subs 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f4138840 con 0x7f47f414be90 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.802+0000 7f47fa2b3640 1 librados: init done 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.803+0000 7f47fa2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.803+0000 7f47fa2b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:07.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.803+0000 7f47fa2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:07.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:07.803+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f40627b0 con 0x7f47f414be90 2024-03-25T18:08:08.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47ec003c20 con 0x7f47f414be90 2024-03-25T18:08:08.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f47ec003ff0 con 0x7f47f414be90 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f0ff9640 10 monclient: dump: 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:08.140 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:08.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.139+0000 7f47f27fc640 4 set_mon_vals no callback set 2024-03-25T18:08:08.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.140+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f47ec004210 con 0x7f47f414be90 2024-03-25T18:08:08.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.141+0000 7f47f3fff640 10 monclient: get_auth_request con 0x7f47e006e0f0 auth_method 0 2024-03-25T18:08:08.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.141+0000 7f47f0ff9640 10 client.4356.objecter ms_handle_connect 0x7f47e006e0f0 2024-03-25T18:08:08.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f47ec054af0 con 0x7f47f414be90 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 10 client.4356.objecter ms_dispatch 0x7f47f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47ec050600 con 0x7f47f414be90 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:08.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.146+0000 7f47f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:08.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.406+0000 7f47fa2b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:08.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.406+0000 7f47fa2b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:08.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.406+0000 7f47fa2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:08.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.406+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47f40b2e00 con 0x7f47f414be90 2024-03-25T18:08:08.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.571+0000 7f47f0ff9640 1 -- 172.21.15.84:0/513143757 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f47ec052530 con 0x7f47f414be90 2024-03-25T18:08:08.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.571+0000 7f47f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:08.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.571+0000 7f47f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:08.572 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:08.572 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 10 librados: watch_flush enter 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 10 librados: watch_flush exit 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 >> v1:172.21.15.157:6800/1801259026 conn(0x7f47e006e0f0 legacy=0x7f47e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 10 monclient: shutdown 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 >> v1:172.21.15.84:6789/0 conn(0x7f47f414be90 legacy=0x7f47f4144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 shutdown_connections 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 >> 172.21.15.84:0/513143757 conn(0x7f47f40617a0 msgr2=0x7f47f40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:08.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 shutdown_connections 2024-03-25T18:08:08.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 -- 172.21.15.84:0/513143757 wait complete. 2024-03-25T18:08:08.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:08.573+0000 7f47fa2b3640 1 librados: shutdown 2024-03-25T18:08:08.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:15.391 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:15.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.613+0000 7f12691c6640 10 monclient: get_monmap_and_config 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.613+0000 7f12691c6640 10 monclient: build_initial_monmap 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.613+0000 7f12691c6640 10 monclient: monmap: 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:15.614289+0000 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:15.614289+0000 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:15.614 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:15.615 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.614+0000 7f12691c6640 1 Processor -- start 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.614+0000 7f12691c6640 1 -- start start 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.614+0000 7f12691c6640 10 monclient: init 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-a con 0x7f12640b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-b con 0x7f12640b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:15.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-c con 0x7f126414be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:15.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640af940 con 0x7f12640b4d60 2024-03-25T18:08:15.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b0a70 con 0x7f126414be90 2024-03-25T18:08:15.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b0d50 con 0x7f12640b1730 2024-03-25T18:08:15.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:15.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.615+0000 7f12691c6640 10 monclient(hunting): authenticate will time out at 7139.913252s 2024-03-25T18:08:15.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.621+0000 7f126359e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f126414be90 0x7f126414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:38222/0 (socket says 172.21.15.84:38222) 2024-03-25T18:08:15.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.621+0000 7f126359e640 1 -- 172.21.15.84:0/1388796417 learned_addr learned my addr 172.21.15.84:0/1388796417 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459010390 0 0) 0x7f12640b0a70 con 0x7f126414be90 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 10 monclient(hunting): _negotiate 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:08:15.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.629+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250003520 con 0x7f126414be90 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491847032 0 0) 0x7f1250003520 con 0x7f126414be90 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 >> v1:172.21.15.84:6790/0 conn(0x7f12640b1730 legacy=0x7f12640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 >> v1:172.21.15.84:6789/0 conn(0x7f12640b4d60 legacy=0x7f126413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: _finish_hunting 0 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 monclient: found mon.noname-c 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640b1030 con 0x7f126414be90 2024-03-25T18:08:15.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: _finish_auth 0 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: _check_auth_tickets 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f125c001a10 con 0x7f126414be90 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f126259c640 10 monclient: dump: 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:15.643 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:15.644 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:15.644 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f12691c6640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:08:15.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.642+0000 7f12691c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f125c001b70 con 0x7f126414be90 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 1 -- 172.21.15.84:0/1388796417 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f125c001f80 con 0x7f126414be90 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f12691c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f12691c6640 4 set_mon_vals no callback set 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.652+0000 7f126259c640 10 monclient: dump: 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.653 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:15.654 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:15.654 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:15.654 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:15.654 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:15.654 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.653+0000 7f12691c6640 10 monclient: shutdown 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.653+0000 7f12691c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.653+0000 7f12691c6640 1 -- 172.21.15.84:0/1388796417 >> v1:172.21.15.157:6789/0 conn(0x7f126414be90 legacy=0x7f126414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.653+0000 7f12691c6640 1 -- 172.21.15.84:0/1388796417 shutdown_connections 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.653+0000 7f12691c6640 1 -- 172.21.15.84:0/1388796417 >> 172.21.15.84:0/1388796417 conn(0x7f12640617a0 msgr2=0x7f1264061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.654+0000 7f12691c6640 1 -- 172.21.15.84:0/1388796417 shutdown_connections 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.654+0000 7f12691c6640 1 -- 172.21.15.84:0/1388796417 wait complete. 2024-03-25T18:08:15.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 10 monclient: build_initial_monmap 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 10 monclient: monmap: 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 librados: starting msgr at 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 librados: starting objecter 2024-03-25T18:08:15.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 Processor -- start 2024-03-25T18:08:15.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 -- start start 2024-03-25T18:08:15.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 librados: setting wanted keys 2024-03-25T18:08:15.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 1 librados: calling monclient init 2024-03-25T18:08:15.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.655+0000 7f12691c6640 10 monclient: init 2024-03-25T18:08:15.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-a con 0x7f12640b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-c con 0x7f12640b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient(hunting): picked mon.noname-b con 0x7f126414be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12641d75d0 con 0x7f12640b1730 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1264141e10 con 0x7f126414be90 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1264142080 con 0x7f12640b4d60 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:15.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.656+0000 7f12691c6640 10 monclient(hunting): authenticate will time out at 7139.954569s 2024-03-25T18:08:15.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.662+0000 7f126359e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f126414be90 0x7f12641daf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:38268/0 (socket says 172.21.15.84:38268) 2024-03-25T18:08:15.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.662+0000 7f126359e640 1 -- 172.21.15.84:0/4002430110 learned_addr learned my addr 172.21.15.84:0/4002430110 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:15.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.664+0000 7f125b7fe640 10 client.?.objecter ms_handle_connect 0x7f126414be90 2024-03-25T18:08:15.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.664+0000 7f125b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:15.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.665+0000 7f125b7fe640 10 client.?.objecter ms_handle_connect 0x7f12640b4d60 2024-03-25T18:08:15.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.665+0000 7f125b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:15.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.669+0000 7f125b7fe640 10 client.?.objecter ms_handle_connect 0x7f12640b1730 2024-03-25T18:08:15.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.669+0000 7f125b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696588685 0 0) 0x7f1264141e10 con 0x7f126414be90 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:08:15.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.674+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1244004eb0 con 0x7f126414be90 2024-03-25T18:08:15.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.681+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778953842 0 0) 0x7f1264142080 con 0x7f12640b4d60 2024-03-25T18:08:15.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.681+0000 7f125b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:15.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.681+0000 7f125b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:15.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.681+0000 7f125b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:15.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.681+0000 7f125b7fe640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:08:15.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.682+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1264141e10 con 0x7f12640b4d60 2024-03-25T18:08:15.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014524314 0 0) 0x7f1244004eb0 con 0x7f126414be90 2024-03-25T18:08:15.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1264142080 con 0x7f126414be90 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f125c001ec0 con 0x7f126414be90 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 10 monclient(hunting): dump: 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:15.689 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:15.690 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.688+0000 7f125b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:15.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850032132 0 0) 0x7f12641d75d0 con 0x7f12640b1730 2024-03-25T18:08:15.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:15.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:15.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:15.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:08:15.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.689+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1244004eb0 con 0x7f12640b1730 2024-03-25T18:08:15.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54125449 0 0) 0x7f1264141e10 con 0x7f12640b4d60 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12641d75d0 con 0x7f12640b4d60 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f124c0018b0 con 0x7f12640b4d60 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.699+0000 7f125b7fe640 10 monclient(hunting): dump: 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:15.700 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:15.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2866832185 0 0) 0x7f1264142080 con 0x7f126414be90 2024-03-25T18:08:15.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 >> v1:172.21.15.84:6790/0 conn(0x7f12640b4d60 legacy=0x7f12641d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 >> v1:172.21.15.84:6789/0 conn(0x7f12640b1730 legacy=0x7f12641405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:15.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 1 monclient: found mon.b 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264140d00 con 0x7f126414be90 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f125b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:08:15.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 10 monclient: _renew_subs 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1264135a90 con 0x7f126414be90 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 10 monclient: _renew_subs 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12641d83a0 con 0x7f126414be90 2024-03-25T18:08:15.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.702+0000 7f12691c6640 1 librados: init done 2024-03-25T18:08:15.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.703+0000 7f12691c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:15.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.703+0000 7f12691c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:15.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.703+0000 7f12691c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:15.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:15.703+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12640b22c0 con 0x7f126414be90 2024-03-25T18:08:16.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f125c0020e0 con 0x7f126414be90 2024-03-25T18:08:16.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f125c004fb0 con 0x7f126414be90 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f1261d9b640 4 set_mon_vals no callback set 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.000+0000 7f125b7fe640 10 monclient: dump: 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:16.001 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:16.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.001+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f125c01af80 con 0x7f126414be90 2024-03-25T18:08:16.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.002+0000 7f1262d9d640 10 monclient: get_auth_request con 0x7f124406df70 auth_method 0 2024-03-25T18:08:16.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.002+0000 7f125b7fe640 10 client.4363.objecter ms_handle_connect 0x7f124406df70 2024-03-25T18:08:16.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.006+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f125c054ae0 con 0x7f126414be90 2024-03-25T18:08:16.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.006+0000 7f125b7fe640 10 client.4363.objecter ms_dispatch 0x7f1264000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:16.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.006+0000 7f125b7fe640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:16.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.006+0000 7f125b7fe640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:16.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.006+0000 7f125b7fe640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T18:08:16.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.007+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f125c050420 con 0x7f126414be90 2024-03-25T18:08:16.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.007+0000 7f125b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:16.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.007+0000 7f125b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:16.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.242+0000 7f12691c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:16.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.242+0000 7f12691c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:16.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.242+0000 7f12691c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:16.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.242+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1264140e90 con 0x7f126414be90 2024-03-25T18:08:16.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.274+0000 7f125b7fe640 1 -- 172.21.15.84:0/4002430110 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f125c052520 con 0x7f126414be90 2024-03-25T18:08:16.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.274+0000 7f125b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:16.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.274+0000 7f125b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:16.275 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:16.275 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:16.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.276+0000 7f12691c6640 10 librados: watch_flush enter 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 10 librados: watch_flush exit 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 10 client.4363.objecter successfully canceled tick 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 >> v1:172.21.15.157:6800/1801259026 conn(0x7f124406df70 legacy=0x7f1244070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 10 monclient: shutdown 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 >> v1:172.21.15.157:6789/0 conn(0x7f126414be90 legacy=0x7f12641daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 shutdown_connections 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 >> 172.21.15.84:0/4002430110 conn(0x7f12640617a0 msgr2=0x7f12640b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 shutdown_connections 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 -- 172.21.15.84:0/4002430110 wait complete. 2024-03-25T18:08:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:16.277+0000 7f12691c6640 1 librados: shutdown 2024-03-25T18:08:16.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:17.759 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:14.578 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.767 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:14.586 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.796 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:14.615 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.817 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:14.601 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.818 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:14.637 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.832 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:14.616 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.836 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:14.655 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.860 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:14.644 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:17.864 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:14.683 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:23.095 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.193+0000 7fd726f58640 10 monclient: get_monmap_and_config 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.193+0000 7fd726f58640 10 monclient: build_initial_monmap 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.193+0000 7fd726f58640 10 monclient: monmap: 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:23.195267+0000 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:23.195267+0000 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.195 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:23.196 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:23.196 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:23.196 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.194+0000 7fd726f58640 1 Processor -- start 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.194+0000 7fd726f58640 1 -- start start 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.194+0000 7fd726f58640 10 monclient: init 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-b con 0x7fd7200b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-c con 0x7fd7200b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-a con 0x7fd72014be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:23.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200af940 con 0x7fd72014be90 2024-03-25T18:08:23.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200b0a70 con 0x7fd7200b1730 2024-03-25T18:08:23.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200b0d50 con 0x7fd7200b4d60 2024-03-25T18:08:23.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:23.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.195+0000 7fd726f58640 10 monclient(hunting): authenticate will time out at 7147.494307s 2024-03-25T18:08:23.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.207+0000 7fd71ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd7200b1730 0x7fd7200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34782/0 (socket says 172.21.15.84:34782) 2024-03-25T18:08:23.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.207+0000 7fd71ffff640 1 -- 172.21.15.84:0/3003903076 learned_addr learned my addr 172.21.15.84:0/3003903076 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:23.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15780036 0 0) 0x7fd7200b0a70 con 0x7fd7200b1730 2024-03-25T18:08:23.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:08:23.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.220+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd710003520 con 0x7fd7200b1730 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3766816525 0 0) 0x7fd710003520 con 0x7fd7200b1730 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 >> v1:172.21.15.84:6790/0 conn(0x7fd7200b4d60 legacy=0x7fd72013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 >> v1:172.21.15.84:6789/0 conn(0x7fd72014be90 legacy=0x7fd72014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:08:23.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7200b1030 con 0x7fd7200b1730 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd70c001690 con 0x7fd7200b1730 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd726f58640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd726f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd71f7fe640 10 monclient: dump: 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.242 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.240+0000 7fd726f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd70c001860 con 0x7fd7200b1730 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 1 -- 172.21.15.84:0/3003903076 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd70c001cb0 con 0x7fd7200b1730 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 4 set_mon_vals no callback set 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:23.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd71f7fe640 10 monclient: dump: 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:23.255 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 10 monclient: shutdown 2024-03-25T18:08:23.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 1 -- 172.21.15.84:0/3003903076 >> v1:172.21.15.157:6789/0 conn(0x7fd7200b1730 legacy=0x7fd7200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 1 -- 172.21.15.84:0/3003903076 shutdown_connections 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.253+0000 7fd726f58640 1 -- 172.21.15.84:0/3003903076 >> 172.21.15.84:0/3003903076 conn(0x7fd7200617a0 msgr2=0x7fd720061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.254+0000 7fd726f58640 1 -- 172.21.15.84:0/3003903076 shutdown_connections 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.254+0000 7fd726f58640 1 -- 172.21.15.84:0/3003903076 wait complete. 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.254+0000 7fd726f58640 10 monclient: build_initial_monmap 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 10 monclient: monmap: 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:23.258 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 librados: starting msgr at 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 librados: starting objecter 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 Processor -- start 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 -- start start 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 librados: setting wanted keys 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 1 librados: calling monclient init 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.255+0000 7fd726f58640 10 monclient: init 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:23.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-c con 0x7fd7200b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-b con 0x7fd7200b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient(hunting): picked mon.noname-a con 0x7fd72014be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7201dadd0 con 0x7fd72014be90 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7201381a0 con 0x7fd7200b4d60 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd720143900 con 0x7fd7200b1730 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:23.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.256+0000 7fd726f58640 10 monclient(hunting): authenticate will time out at 7147.555156s 2024-03-25T18:08:23.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.263+0000 7fd71ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd7200b4d60 0x7fd7201d7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34828/0 (socket says 172.21.15.84:34828) 2024-03-25T18:08:23.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.263+0000 7fd71ffff640 1 -- 172.21.15.84:0/1127572238 learned_addr learned my addr 172.21.15.84:0/1127572238 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:23.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.266+0000 7fd71d7fa640 10 client.?.objecter ms_handle_connect 0x7fd72014be90 2024-03-25T18:08:23.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.266+0000 7fd71d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.267+0000 7fd71d7fa640 10 client.?.objecter ms_handle_connect 0x7fd7200b4d60 2024-03-25T18:08:23.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.267+0000 7fd71d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.278+0000 7fd71d7fa640 10 client.?.objecter ms_handle_connect 0x7fd7200b1730 2024-03-25T18:08:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.278+0000 7fd71d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578719893 0 0) 0x7fd7201381a0 con 0x7fd7200b4d60 2024-03-25T18:08:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:08:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.279+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704004eb0 con 0x7fd7200b4d60 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178378081 0 0) 0x7fd7201dadd0 con 0x7fd72014be90 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:08:23.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.291+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7201381a0 con 0x7fd72014be90 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.294+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409967299 0 0) 0x7fd704004eb0 con 0x7fd7200b4d60 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7201dadd0 con 0x7fd7200b4d60 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd70c001b70 con 0x7fd7200b4d60 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 10 monclient(hunting): dump: 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.296 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.295+0000 7fd71d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964949527 0 0) 0x7fd720143900 con 0x7fd7200b1730 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:08:23.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.297+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704004eb0 con 0x7fd7200b1730 2024-03-25T18:08:23.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 856548632 0 0) 0x7fd7201381a0 con 0x7fd72014be90 2024-03-25T18:08:23.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd720143900 con 0x7fd72014be90 2024-03-25T18:08:23.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd718001a70 con 0x7fd72014be90 2024-03-25T18:08:23.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.315+0000 7fd71d7fa640 10 monclient(hunting): dump: 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 339080574 0 0) 0x7fd7201dadd0 con 0x7fd7200b4d60 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 >> v1:172.21.15.84:6790/0 conn(0x7fd7200b1730 legacy=0x7fd7201db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 >> v1:172.21.15.84:6789/0 conn(0x7fd72014be90 legacy=0x7fd7201d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 1 monclient: found mon.b 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd720143a90 con 0x7fd7200b4d60 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd71d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 10 monclient: _renew_subs 2024-03-25T18:08:23.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7201db970 con 0x7fd7200b4d60 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 10 monclient: _renew_subs 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7201e3430 con 0x7fd7200b4d60 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.316+0000 7fd726f58640 1 librados: init done 2024-03-25T18:08:23.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.317+0000 7fd726f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.317+0000 7fd726f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.317+0000 7fd726f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.317+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7201401f0 con 0x7fd7200b4d60 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd70c004bf0 con 0x7fd7200b4d60 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd70c004f00 con 0x7fd7200b4d60 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71d7fa640 10 monclient: dump: 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:23.633 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:23.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.632+0000 7fd71effd640 4 set_mon_vals no callback set 2024-03-25T18:08:23.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.633+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fd70c01aed0 con 0x7fd7200b4d60 2024-03-25T18:08:23.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.633+0000 7fd724cf4640 10 monclient: get_auth_request con 0x7fd70406df60 auth_method 0 2024-03-25T18:08:23.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.634+0000 7fd71d7fa640 10 client.4369.objecter ms_handle_connect 0x7fd70406df60 2024-03-25T18:08:23.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.638+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fd70c054960 con 0x7fd7200b4d60 2024-03-25T18:08:23.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.638+0000 7fd71d7fa640 10 client.4369.objecter ms_dispatch 0x7fd720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:23.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.638+0000 7fd71d7fa640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:23.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.638+0000 7fd71d7fa640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:23.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.638+0000 7fd71d7fa640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T18:08:23.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.639+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd70c050470 con 0x7fd7200b4d60 2024-03-25T18:08:23.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.639+0000 7fd71d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.639+0000 7fd71d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:23.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.893+0000 7fd726f58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.893+0000 7fd726f58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.893+0000 7fd726f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:08:23.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.893+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7201daf80 con 0x7fd7200b4d60 2024-03-25T18:08:23.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.913+0000 7fd71d7fa640 1 -- 172.21.15.84:0/1127572238 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fd7201daf80 con 0x7fd7200b4d60 2024-03-25T18:08:23.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.913+0000 7fd71d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.913+0000 7fd71d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:23.914 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:23.914 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:23.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 10 librados: watch_flush enter 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 10 librados: watch_flush exit 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 10 client.4369.objecter successfully canceled tick 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 >> v1:172.21.15.157:6800/1801259026 conn(0x7fd70406df60 legacy=0x7fd704070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 10 monclient: shutdown 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 >> v1:172.21.15.157:6789/0 conn(0x7fd7200b4d60 legacy=0x7fd7201d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 shutdown_connections 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 >> 172.21.15.84:0/1127572238 conn(0x7fd7200617a0 msgr2=0x7fd7200b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 shutdown_connections 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 -- 172.21.15.84:0/1127572238 wait complete. 2024-03-25T18:08:23.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:23.915+0000 7fd726f58640 1 librados: shutdown 2024-03-25T18:08:23.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:30.735 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:30.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.853+0000 7f36c0281640 10 monclient: get_monmap_and_config 2024-03-25T18:08:30.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.853+0000 7f36c0281640 10 monclient: build_initial_monmap 2024-03-25T18:08:30.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.853+0000 7f36c0281640 10 monclient: monmap: 2024-03-25T18:08:30.854 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:30.854 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:30.854624+0000 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:30.854624+0000 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:30.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.853+0000 7f36c0281640 1 Processor -- start 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.854+0000 7f36c0281640 1 -- start start 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.854+0000 7f36c0281640 10 monclient: init 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-a con 0x7f36b80a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-b con 0x7f36b813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:30.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-c con 0x7f36b81430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:30.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b8142520 con 0x7f36b80a98f0 2024-03-25T18:08:30.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b80b28a0 con 0x7f36b81430c0 2024-03-25T18:08:30.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b80b2a30 con 0x7f36b813f710 2024-03-25T18:08:30.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.855+0000 7f36c0281640 10 monclient(hunting): authenticate will time out at 7155.154302s 2024-03-25T18:08:30.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.859+0000 7f36be81e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f36b81430c0 0x7f36b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:33542/0 (socket says 172.21.15.84:33542) 2024-03-25T18:08:30.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.859+0000 7f36be81e640 1 -- 172.21.15.84:0/1248788228 learned_addr learned my addr 172.21.15.84:0/1248788228 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224442350 0 0) 0x7f36b80b28a0 con 0x7f36b81430c0 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:08:30.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.866+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a4003520 con 0x7f36b81430c0 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833159396 0 0) 0x7f36b80b2a30 con 0x7f36b813f710 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:08:30.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.877+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b80b28a0 con 0x7f36b813f710 2024-03-25T18:08:30.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646880761 0 0) 0x7f36a4003520 con 0x7f36b81430c0 2024-03-25T18:08:30.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 >> v1:172.21.15.84:6790/0 conn(0x7f36b813f710 legacy=0x7f36b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 >> v1:172.21.15.84:6789/0 conn(0x7f36b80a98f0 legacy=0x7f36b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: _finish_hunting 0 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 monclient: found mon.noname-c 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80b44c0 con 0x7f36b81430c0 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: _finish_auth 0 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: _check_auth_tickets 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f36b0001a10 con 0x7f36b81430c0 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36bd01b640 10 monclient: dump: 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.880 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36c0281640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:08:30.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.878+0000 7f36c0281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:30.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b0001b70 con 0x7f36b81430c0 2024-03-25T18:08:30.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:30.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 1 -- 172.21.15.84:0/1248788228 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f36b0001f80 con 0x7f36b81430c0 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36c0281640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36c0281640 4 set_mon_vals no callback set 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.886+0000 7f36bd01b640 10 monclient: dump: 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:30.888 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:30.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 10 monclient: shutdown 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 1 -- 172.21.15.84:0/1248788228 >> v1:172.21.15.157:6789/0 conn(0x7f36b81430c0 legacy=0x7f36b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 1 -- 172.21.15.84:0/1248788228 shutdown_connections 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 1 -- 172.21.15.84:0/1248788228 >> 172.21.15.84:0/1248788228 conn(0x7f36b80b7ea0 msgr2=0x7f36b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 1 -- 172.21.15.84:0/1248788228 shutdown_connections 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.887+0000 7f36c0281640 1 -- 172.21.15.84:0/1248788228 wait complete. 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 10 monclient: build_initial_monmap 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 10 monclient: monmap: 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:30.891 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 librados: starting msgr at 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 librados: starting objecter 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 Processor -- start 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 -- start start 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 librados: setting wanted keys 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 1 librados: calling monclient init 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.888+0000 7f36c0281640 10 monclient: init 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-a con 0x7f36b80a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-b con 0x7f36b813f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient(hunting): picked mon.noname-c con 0x7f36b81430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b81369d0 con 0x7f36b80a98f0 2024-03-25T18:08:30.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b8133a60 con 0x7f36b813f710 2024-03-25T18:08:30.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b80ab9e0 con 0x7f36b81430c0 2024-03-25T18:08:30.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.889+0000 7f36c0281640 10 monclient(hunting): authenticate will time out at 7155.188551s 2024-03-25T18:08:30.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.898+0000 7f36be81e640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f36b81430c0 0x7f36b8133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34430/0 (socket says 172.21.15.84:34430) 2024-03-25T18:08:30.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.898+0000 7f36be81e640 1 -- 172.21.15.84:0/128088961 learned_addr learned my addr 172.21.15.84:0/128088961 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:30.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.899+0000 7f36aeffd640 10 client.?.objecter ms_handle_connect 0x7f36b81430c0 2024-03-25T18:08:30.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.899+0000 7f36aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.905+0000 7f36aeffd640 10 client.?.objecter ms_handle_connect 0x7f36b813f710 2024-03-25T18:08:30.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.905+0000 7f36aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.906+0000 7f36aeffd640 10 client.?.objecter ms_handle_connect 0x7f36b80a98f0 2024-03-25T18:08:30.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.906+0000 7f36aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586905183 0 0) 0x7f36b80ab9e0 con 0x7f36b81430c0 2024-03-25T18:08:30.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:08:30.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.911+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c004eb0 con 0x7f36b81430c0 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329800305 0 0) 0x7f36b8133a60 con 0x7f36b813f710 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:08:30.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.915+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b80ab9e0 con 0x7f36b813f710 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847816939 0 0) 0x7f369c004eb0 con 0x7f36b81430c0 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b8133a60 con 0x7f36b81430c0 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f36b0001ec0 con 0x7f36b81430c0 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 10 monclient(hunting): dump: 2024-03-25T18:08:30.921 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:30.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.920+0000 7f36aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291845683 0 0) 0x7f36b8133a60 con 0x7f36b81430c0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 >> v1:172.21.15.157:6789/0 conn(0x7f36b813f710 legacy=0x7f36b81362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 >> v1:172.21.15.84:6789/0 conn(0x7f36b80a98f0 legacy=0x7f36b8132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 1 monclient: found mon.c 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80abcc0 con 0x7f36b81430c0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:30.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.927+0000 7f36c0281640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 10 monclient: _renew_subs 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b80abfa0 con 0x7f36b81430c0 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 10 monclient: _renew_subs 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b8137e40 con 0x7f36b81430c0 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.928+0000 7f36c0281640 1 librados: init done 2024-03-25T18:08:30.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.929+0000 7f36c0281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:30.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.929+0000 7f36c0281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:30.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.929+0000 7f36c0281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:30.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:30.929+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b80627b0 con 0x7f36b81430c0 2024-03-25T18:08:31.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b0004e10 con 0x7f36b81430c0 2024-03-25T18:08:31.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f36b0002100 con 0x7f36b81430c0 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36aeffd640 10 monclient: dump: 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:31.235 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:31.236 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:31.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.233+0000 7f36bc81a640 4 set_mon_vals no callback set 2024-03-25T18:08:31.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.234+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f36b001af90 con 0x7f36b81430c0 2024-03-25T18:08:31.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.234+0000 7f36be01d640 10 monclient: get_auth_request con 0x7f369c06dd10 auth_method 0 2024-03-25T18:08:31.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.235+0000 7f36aeffd640 10 client.4376.objecter ms_handle_connect 0x7f369c06dd10 2024-03-25T18:08:31.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f36b0054b40 con 0x7f36b81430c0 2024-03-25T18:08:31.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 10 client.4376.objecter ms_dispatch 0x7f36b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:31.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:31.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:31.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T18:08:31.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36b0050480 con 0x7f36b81430c0 2024-03-25T18:08:31.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:31.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.239+0000 7f36aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:31.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.483+0000 7f36c0281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.483+0000 7f36c0281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.483+0000 7f36c0281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:31.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.483+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b8139000 con 0x7f36b81430c0 2024-03-25T18:08:31.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.501+0000 7f36aeffd640 1 -- 172.21.15.84:0/128088961 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f36b0052580 con 0x7f36b81430c0 2024-03-25T18:08:31.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.501+0000 7f36aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.501+0000 7f36aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:31.502 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:31.502 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:31.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.503+0000 7f36c0281640 10 librados: watch_flush enter 2024-03-25T18:08:31.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.503+0000 7f36c0281640 10 librados: watch_flush exit 2024-03-25T18:08:31.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T18:08:31.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 10 client.4376.objecter successfully canceled tick 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 >> v1:172.21.15.157:6800/1801259026 conn(0x7f369c06dd10 legacy=0x7f369c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 10 monclient: shutdown 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 >> v1:172.21.15.84:6790/0 conn(0x7f36b81430c0 legacy=0x7f36b8133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 shutdown_connections 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 >> 172.21.15.84:0/128088961 conn(0x7f36b80b7ea0 msgr2=0x7f36b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 shutdown_connections 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.504+0000 7f36c0281640 1 -- 172.21.15.84:0/128088961 wait complete. 2024-03-25T18:08:31.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:31.505+0000 7f36c0281640 1 librados: shutdown 2024-03-25T18:08:31.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:37.844 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:34.663 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.850 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:34.669 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.879 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:34.698 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.902 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:34.686 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.910 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:34.694 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.913 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:34.733 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.923 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:34.742 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.946 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:34.730 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:37.957 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:34.776 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:38.324 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:38.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.438+0000 7fb001b3d640 10 monclient: get_monmap_and_config 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.438+0000 7fb001b3d640 10 monclient: build_initial_monmap 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.438+0000 7fb001b3d640 10 monclient: monmap: 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:38.440097+0000 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:38.440097+0000 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:38.440 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.439+0000 7fb001b3d640 1 Processor -- start 2024-03-25T18:08:38.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.439+0000 7fb001b3d640 1 -- start start 2024-03-25T18:08:38.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.439+0000 7fb001b3d640 10 monclient: init 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-a con 0x7faffc0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-c con 0x7faffc13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-b con 0x7faffc1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc142520 con 0x7faffc0a98f0 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc0b28a0 con 0x7faffc13f710 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc0b2a30 con 0x7faffc1430c0 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:38.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.440+0000 7fb001b3d640 10 monclient(hunting): authenticate will time out at 7162.739713s 2024-03-25T18:08:38.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.454+0000 7faffaffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7faffc13f710 0x7faffc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:38432/0 (socket says 172.21.15.84:38432) 2024-03-25T18:08:38.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.454+0000 7faffaffd640 1 -- 172.21.15.84:0/1223078677 learned_addr learned my addr 172.21.15.84:0/1223078677 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269996919 0 0) 0x7faffc0b2a30 con 0x7faffc1430c0 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe4003520 con 0x7faffc1430c0 2024-03-25T18:08:38.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520425735 0 0) 0x7faffc0b28a0 con 0x7faffc13f710 2024-03-25T18:08:38.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:08:38.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.467+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc0b2a30 con 0x7faffc13f710 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740304082 0 0) 0x7faffc142520 con 0x7faffc0a98f0 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:08:38.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.482+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc0b28a0 con 0x7faffc0a98f0 2024-03-25T18:08:38.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413340993 0 0) 0x7faffc0b2a30 con 0x7faffc13f710 2024-03-25T18:08:38.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 >> v1:172.21.15.84:6790/0 conn(0x7faffc1430c0 legacy=0x7faffc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 >> v1:172.21.15.84:6789/0 conn(0x7faffc0a98f0 legacy=0x7faffc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 monclient: found mon.noname-c 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0b44c0 con 0x7faffc13f710 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7faff0001670 con 0x7faffc13f710 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7fb001b3d640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7faffa7fc640 10 monclient: dump: 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:38.487 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.485+0000 7fb001b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faff00017d0 con 0x7faffc13f710 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 1 -- 172.21.15.84:0/1223078677 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7faff0001be0 con 0x7faffc13f710 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7fb001b3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7fb001b3d640 4 set_mon_vals no callback set 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7faffa7fc640 10 monclient: dump: 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:38.513 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:38.514 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7fb001b3d640 10 monclient: shutdown 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7fb001b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.512+0000 7fb001b3d640 1 -- 172.21.15.84:0/1223078677 >> v1:172.21.15.157:6789/0 conn(0x7faffc13f710 legacy=0x7faffc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.513+0000 7fb001b3d640 1 -- 172.21.15.84:0/1223078677 shutdown_connections 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.513+0000 7fb001b3d640 1 -- 172.21.15.84:0/1223078677 >> 172.21.15.84:0/1223078677 conn(0x7faffc0b7ea0 msgr2=0x7faffc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.513+0000 7fb001b3d640 1 -- 172.21.15.84:0/1223078677 shutdown_connections 2024-03-25T18:08:38.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.513+0000 7fb001b3d640 1 -- 172.21.15.84:0/1223078677 wait complete. 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 10 monclient: build_initial_monmap 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 10 monclient: monmap: 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 librados: starting msgr at 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 librados: starting objecter 2024-03-25T18:08:38.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 Processor -- start 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 -- start start 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 librados: setting wanted keys 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 1 librados: calling monclient init 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.514+0000 7fb001b3d640 10 monclient: init 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-b con 0x7faffc0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-a con 0x7faffc13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient(hunting): picked mon.noname-c con 0x7faffc1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc136510 con 0x7faffc13f710 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc0abb00 con 0x7faffc0a98f0 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc1da990 con 0x7faffc1430c0 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:38.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.515+0000 7fb001b3d640 10 monclient(hunting): authenticate will time out at 7162.814736s 2024-03-25T18:08:38.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.518+0000 7faffbfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7faffc1430c0 0x7faffc1e1c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35852/0 (socket says 172.21.15.84:35852) 2024-03-25T18:08:38.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.518+0000 7faffbfff640 1 -- 172.21.15.84:0/3302065380 learned_addr learned my addr 172.21.15.84:0/3302065380 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:38.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.523+0000 7fafe3fff640 10 client.?.objecter ms_handle_connect 0x7faffc1430c0 2024-03-25T18:08:38.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.523+0000 7fafe3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.530+0000 7fafe3fff640 10 client.?.objecter ms_handle_connect 0x7faffc13f710 2024-03-25T18:08:38.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.530+0000 7fafe3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.540+0000 7fafe3fff640 10 client.?.objecter ms_handle_connect 0x7faffc0a98f0 2024-03-25T18:08:38.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.540+0000 7fafe3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035493529 0 0) 0x7faffc136510 con 0x7faffc13f710 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:08:38.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.542+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8004eb0 con 0x7faffc13f710 2024-03-25T18:08:38.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900645535 0 0) 0x7faffc1da990 con 0x7faffc1430c0 2024-03-25T18:08:38.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:08:38.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.543+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc136510 con 0x7faffc1430c0 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992986586 0 0) 0x7faffc0abb00 con 0x7faffc0a98f0 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:08:38.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.552+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc1da990 con 0x7faffc0a98f0 2024-03-25T18:08:38.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.556+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547585969 0 0) 0x7faffc136510 con 0x7faffc1430c0 2024-03-25T18:08:38.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.556+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faffc0abb00 con 0x7faffc1430c0 2024-03-25T18:08:38.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.557+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7faff4001c10 con 0x7faffc1430c0 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.557+0000 7fafe3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.557+0000 7fafe3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.557+0000 7fafe3fff640 10 monclient(hunting): dump: 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.557+0000 7fafe3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3377977724 0 0) 0x7faffc0abb00 con 0x7faffc1430c0 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 >> v1:172.21.15.157:6789/0 conn(0x7faffc0a98f0 legacy=0x7faffc1daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 >> v1:172.21.15.84:6789/0 conn(0x7faffc13f710 legacy=0x7faffc1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 1 monclient: found mon.c 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc137820 con 0x7faffc1430c0 2024-03-25T18:08:38.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fafe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 5 monclient: authenticate success, global_id 4382 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 10 monclient: _renew_subs 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc1db5c0 con 0x7faffc1430c0 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 10 monclient: _renew_subs 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc138c90 con 0x7faffc1430c0 2024-03-25T18:08:38.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.563+0000 7fb001b3d640 1 librados: init done 2024-03-25T18:08:38.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.564+0000 7fb001b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.564+0000 7fb001b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.564+0000 7fb001b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:38.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.564+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc0b3ef0 con 0x7faffc1430c0 2024-03-25T18:08:38.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faff40020e0 con 0x7faffc1430c0 2024-03-25T18:08:38.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7faff4003b10 con 0x7faffc1430c0 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7fafe3fff640 10 monclient: dump: 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:38.870 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:38.871 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:38.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.868+0000 7faff9ffb640 4 set_mon_vals no callback set 2024-03-25T18:08:38.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.869+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7faff4003d30 con 0x7faffc1430c0 2024-03-25T18:08:38.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.870+0000 7faffb7fe640 10 monclient: get_auth_request con 0x7fafe806dda0 auth_method 0 2024-03-25T18:08:38.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.870+0000 7fafe3fff640 10 client.4382.objecter ms_handle_connect 0x7fafe806dda0 2024-03-25T18:08:38.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7faff4054a20 con 0x7faffc1430c0 2024-03-25T18:08:38.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 10 client.4382.objecter ms_dispatch 0x7faffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:38.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:38.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:38.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T18:08:38.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faff4050530 con 0x7faffc1430c0 2024-03-25T18:08:38.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:38.876+0000 7fafe3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.138+0000 7fb001b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.138+0000 7fb001b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.138+0000 7fb001b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.138+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faffc14f320 con 0x7faffc1430c0 2024-03-25T18:08:39.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.171+0000 7fafe3fff640 1 -- 172.21.15.84:0/3302065380 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7faff4052460 con 0x7faffc1430c0 2024-03-25T18:08:39.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.171+0000 7fafe3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.171+0000 7fafe3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:39.172 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:39.173 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:39.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.173+0000 7fb001b3d640 10 librados: watch_flush enter 2024-03-25T18:08:39.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.173+0000 7fb001b3d640 10 librados: watch_flush exit 2024-03-25T18:08:39.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.173+0000 7fb001b3d640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T18:08:39.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.173+0000 7fb001b3d640 10 client.4382.objecter successfully canceled tick 2024-03-25T18:08:39.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 >> v1:172.21.15.157:6800/1801259026 conn(0x7fafe806dda0 legacy=0x7fafe8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 10 monclient: shutdown 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 >> v1:172.21.15.84:6790/0 conn(0x7faffc1430c0 legacy=0x7faffc1e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 shutdown_connections 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 >> 172.21.15.84:0/3302065380 conn(0x7faffc0b7ea0 msgr2=0x7faffc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 shutdown_connections 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 -- 172.21.15.84:0/3302065380 wait complete. 2024-03-25T18:08:39.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:39.174+0000 7fb001b3d640 1 librados: shutdown 2024-03-25T18:08:39.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:45.994 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.109+0000 7fd922a4e640 10 monclient: get_monmap_and_config 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.109+0000 7fd922a4e640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.110+0000 7fd922a4e640 10 monclient: monmap: 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:46.111392+0000 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:46.111392+0000 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:46.111 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:46.112 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:46.112 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:46.112 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.110+0000 7fd922a4e640 1 Processor -- start 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.110+0000 7fd922a4e640 1 -- start start 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.110+0000 7fd922a4e640 10 monclient: init 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd91c06adf0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd91c13daa0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd91c0af580 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0b0d70 con 0x7fd91c06adf0 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0b1050 con 0x7fd91c13daa0 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0b1330 con 0x7fd91c0af580 2024-03-25T18:08:46.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.111+0000 7fd922a4e640 10 monclient(hunting): authenticate will time out at 7170.410712s 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.121+0000 7fd91bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fd91c06adf0 0x7fd91c13d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:38786/0 (socket says 172.21.15.84:38786) 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.121+0000 7fd91bfff640 1 -- 172.21.15.84:0/128176886 learned_addr learned my addr 172.21.15.84:0/128176886 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050561859 0 0) 0x7fd91c0b0d70 con 0x7fd91c06adf0 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:08:46.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.140+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904003520 con 0x7fd91c06adf0 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210266247 0 0) 0x7fd91c0b1050 con 0x7fd91c13daa0 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:08:46.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.153+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c0b0d70 con 0x7fd91c13daa0 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128942679 0 0) 0x7fd91c0b1330 con 0x7fd91c0af580 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:08:46.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.166+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c0b1050 con 0x7fd91c0af580 2024-03-25T18:08:46.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909350877 0 0) 0x7fd904003520 con 0x7fd91c06adf0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 >> v1:172.21.15.84:6790/0 conn(0x7fd91c0af580 legacy=0x7fd91c1408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 >> v1:172.21.15.157:6789/0 conn(0x7fd91c13daa0 legacy=0x7fd91c1401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 monclient: found mon.noname-a 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0afb10 con 0x7fd91c06adf0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd90c001690 con 0x7fd91c06adf0 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd91affd640 10 monclient: dump: 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd922a4e640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:08:46.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.172+0000 7fd922a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:46.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd90c0017f0 con 0x7fd91c06adf0 2024-03-25T18:08:46.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 1 -- 172.21.15.84:0/128176886 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd90c001c00 con 0x7fd91c06adf0 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd91affd640 10 monclient: dump: 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:46.184 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:46.185 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd922a4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:46.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.182+0000 7fd922a4e640 4 set_mon_vals no callback set 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 10 monclient: shutdown 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 1 -- 172.21.15.84:0/128176886 >> v1:172.21.15.84:6789/0 conn(0x7fd91c06adf0 legacy=0x7fd91c13d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 1 -- 172.21.15.84:0/128176886 shutdown_connections 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 1 -- 172.21.15.84:0/128176886 >> 172.21.15.84:0/128176886 conn(0x7fd91c06c830 msgr2=0x7fd91c0b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 1 -- 172.21.15.84:0/128176886 shutdown_connections 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.183+0000 7fd922a4e640 1 -- 172.21.15.84:0/128176886 wait complete. 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.184+0000 7fd922a4e640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.184+0000 7fd922a4e640 10 monclient: monmap: 2024-03-25T18:08:46.187 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.184+0000 7fd922a4e640 1 librados: starting msgr at 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.184+0000 7fd922a4e640 1 librados: starting objecter 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.184+0000 7fd922a4e640 1 Processor -- start 2024-03-25T18:08:46.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 1 -- start start 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 1 librados: setting wanted keys 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 1 librados: calling monclient init 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient: init 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd91c06adf0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd91c0af580 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd91c13daa0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.185+0000 7fd922a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c1340b0 con 0x7fd91c0af580 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.186+0000 7fd922a4e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c1354d0 con 0x7fd91c13daa0 2024-03-25T18:08:46.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.186+0000 7fd922a4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c136150 con 0x7fd91c06adf0 2024-03-25T18:08:46.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.186+0000 7fd922a4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.186+0000 7fd922a4e640 10 monclient(hunting): authenticate will time out at 7170.484919s 2024-03-25T18:08:46.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.191+0000 7fd91bfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fd91c06adf0 0x7fd91c0a9680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40990/0 (socket says 172.21.15.84:40990) 2024-03-25T18:08:46.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.191+0000 7fd91bfff640 1 -- 172.21.15.84:0/4163953142 learned_addr learned my addr 172.21.15.84:0/4163953142 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:46.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.194+0000 7fd918ff9640 10 client.?.objecter ms_handle_connect 0x7fd91c06adf0 2024-03-25T18:08:46.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.194+0000 7fd918ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.194+0000 7fd918ff9640 10 client.?.objecter ms_handle_connect 0x7fd91c13daa0 2024-03-25T18:08:46.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.194+0000 7fd918ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.198+0000 7fd918ff9640 10 client.?.objecter ms_handle_connect 0x7fd91c0af580 2024-03-25T18:08:46.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.198+0000 7fd918ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880805832 0 0) 0x7fd91c1340b0 con 0x7fd91c0af580 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:08:46.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.206+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd908004eb0 con 0x7fd91c0af580 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609285372 0 0) 0x7fd91c1354d0 con 0x7fd91c13daa0 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:08:46.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.214+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c1340b0 con 0x7fd91c13daa0 2024-03-25T18:08:46.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040981983 0 0) 0x7fd91c136150 con 0x7fd91c06adf0 2024-03-25T18:08:46.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:08:46.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.220+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c1354d0 con 0x7fd91c06adf0 2024-03-25T18:08:46.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 198554752 0 0) 0x7fd908004eb0 con 0x7fd91c0af580 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd91c136150 con 0x7fd91c0af580 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd910001910 con 0x7fd91c0af580 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:46.227 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:46.228 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.225+0000 7fd918ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:46.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1422131332 0 0) 0x7fd91c136150 con 0x7fd91c0af580 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 >> v1:172.21.15.84:6790/0 conn(0x7fd91c06adf0 legacy=0x7fd91c0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 >> v1:172.21.15.157:6789/0 conn(0x7fd91c13daa0 legacy=0x7fd91c132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 1 monclient: found mon.a 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c136430 con 0x7fd91c0af580 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd918ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 10 monclient: _renew_subs 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c136710 con 0x7fd91c0af580 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 10 monclient: _renew_subs 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c1e1280 con 0x7fd91c0af580 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.236+0000 7fd922a4e640 1 librados: init done 2024-03-25T18:08:46.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.237+0000 7fd922a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.237+0000 7fd922a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.237+0000 7fd922a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.237+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c0627b0 con 0x7fd91c0af580 2024-03-25T18:08:46.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9100035f0 con 0x7fd91c0af580 2024-03-25T18:08:46.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fd910003900 con 0x7fd91c0af580 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd918ff9640 10 monclient: dump: 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:46.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.554+0000 7fd91a7fc640 4 set_mon_vals no callback set 2024-03-25T18:08:46.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.555+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fd9100198d0 con 0x7fd91c0af580 2024-03-25T18:08:46.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.555+0000 7fd91bfff640 10 monclient: get_auth_request con 0x7fd90806de80 auth_method 0 2024-03-25T18:08:46.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.556+0000 7fd918ff9640 10 client.4383.objecter ms_handle_connect 0x7fd90806de80 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fd910053510 con 0x7fd91c0af580 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 10 client.4383.objecter ms_dispatch 0x7fd91c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-25T18:08:46.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd91004f020 con 0x7fd91c0af580 2024-03-25T18:08:46.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.560+0000 7fd918ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:46.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.780+0000 7fd922a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.780+0000 7fd922a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.780+0000 7fd922a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:08:46.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.780+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91c0b0000 con 0x7fd91c0af580 2024-03-25T18:08:46.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.797+0000 7fd918ff9640 1 -- 172.21.15.84:0/4163953142 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fd910050f50 con 0x7fd91c0af580 2024-03-25T18:08:46.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.797+0000 7fd918ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.797+0000 7fd918ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:46.799 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:46.799 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:46.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 10 librados: watch_flush enter 2024-03-25T18:08:46.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 10 librados: watch_flush exit 2024-03-25T18:08:46.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-25T18:08:46.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 10 client.4383.objecter successfully canceled tick 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 >> v1:172.21.15.157:6800/1801259026 conn(0x7fd90806de80 legacy=0x7fd908070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 10 monclient: shutdown 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 >> v1:172.21.15.84:6789/0 conn(0x7fd91c0af580 legacy=0x7fd91c134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 shutdown_connections 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 >> 172.21.15.84:0/4163953142 conn(0x7fd91c06c830 msgr2=0x7fd91c13a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.799+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 shutdown_connections 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.800+0000 7fd922a4e640 1 -- 172.21.15.84:0/4163953142 wait complete. 2024-03-25T18:08:46.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:46.800+0000 7fd922a4e640 1 librados: shutdown 2024-03-25T18:08:46.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:53.619 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.736+0000 7f2daa972640 10 monclient: get_monmap_and_config 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.736+0000 7f2daa972640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.736+0000 7f2daa972640 10 monclient: monmap: 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:08:53.737517+0000 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:08:53.737517+0000 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.737 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:53.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:08:53.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:08:53.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.737+0000 7f2daa972640 1 Processor -- start 2024-03-25T18:08:53.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.737+0000 7f2daa972640 1 -- start start 2024-03-25T18:08:53.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.737+0000 7f2daa972640 10 monclient: init 2024-03-25T18:08:53.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:53.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-a con 0x7f2da40705b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:53.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-b con 0x7f2da41402d0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:53.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-c con 0x7f2da4149ae0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:53.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da40718c0 con 0x7f2da40705b0 2024-03-25T18:08:53.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da40714c0 con 0x7f2da4149ae0 2024-03-25T18:08:53.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.738+0000 7f2daa972640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da4072b50 con 0x7f2da41402d0 2024-03-25T18:08:53.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.739+0000 7f2daa972640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.739+0000 7f2daa972640 10 monclient(hunting): authenticate will time out at 7178.037914s 2024-03-25T18:08:53.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.743+0000 7f2da3fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2da40705b0 0x7f2da413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43684/0 (socket says 172.21.15.84:43684) 2024-03-25T18:08:53.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.743+0000 7f2da3fff640 1 -- 172.21.15.84:0/1825587044 learned_addr learned my addr 172.21.15.84:0/1825587044 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446245471 0 0) 0x7f2da40714c0 con 0x7f2da4149ae0 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.752+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8c003520 con 0x7f2da4149ae0 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327832534 0 0) 0x7f2da40718c0 con 0x7f2da40705b0 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:08:53.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.754+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da40714c0 con 0x7f2da40705b0 2024-03-25T18:08:53.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289507867 0 0) 0x7f2da40714c0 con 0x7f2da40705b0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 >> v1:172.21.15.84:6790/0 conn(0x7f2da41402d0 legacy=0x7f2da4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 >> v1:172.21.15.157:6789/0 conn(0x7f2da4149ae0 legacy=0x7f2da414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da413dd10 con 0x7f2da40705b0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d9c0019f0 con 0x7f2da40705b0 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2da2ffd640 10 monclient: dump: 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:53.771 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2daa972640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:08:53.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.769+0000 7f2daa972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d9c001b50 con 0x7f2da40705b0 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 1 -- 172.21.15.84:0/1825587044 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d9c001f60 con 0x7f2da40705b0 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2da2ffd640 10 monclient: dump: 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.785 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2daa972640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:53.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.784+0000 7f2daa972640 4 set_mon_vals no callback set 2024-03-25T18:08:53.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 10 monclient: shutdown 2024-03-25T18:08:53.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:53.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 1 -- 172.21.15.84:0/1825587044 >> v1:172.21.15.84:6789/0 conn(0x7f2da40705b0 legacy=0x7f2da413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 1 -- 172.21.15.84:0/1825587044 shutdown_connections 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 1 -- 172.21.15.84:0/1825587044 >> 172.21.15.84:0/1825587044 conn(0x7f2da40b5d10 msgr2=0x7f2da40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 1 -- 172.21.15.84:0/1825587044 shutdown_connections 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 1 -- 172.21.15.84:0/1825587044 wait complete. 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.785+0000 7f2daa972640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 10 monclient: monmap: 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:08:53.788 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 librados: starting msgr at 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 librados: starting objecter 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 Processor -- start 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 -- start start 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 librados: setting wanted keys 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 1 librados: calling monclient init 2024-03-25T18:08:53.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.786+0000 7f2daa972640 10 monclient: init 2024-03-25T18:08:53.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:53.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-a con 0x7f2da40705b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:08:53.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-b con 0x7f2da41402d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:08:53.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient(hunting): picked mon.noname-c con 0x7f2da4149ae0 addr v1:172.21.15.84:6790/0 2024-03-25T18:08:53.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da4147260 con 0x7f2da40705b0 2024-03-25T18:08:53.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da41db7d0 con 0x7f2da41402d0 2024-03-25T18:08:53.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da4142320 con 0x7f2da4149ae0 2024-03-25T18:08:53.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.787+0000 7f2daa972640 10 monclient(hunting): authenticate will time out at 7178.086525s 2024-03-25T18:08:53.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.801+0000 7f2da8f0f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f2da4149ae0 0x7f2da41458a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60202/0 (socket says 172.21.15.84:60202) 2024-03-25T18:08:53.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.801+0000 7f2da8f0f640 1 -- 172.21.15.84:0/2668502846 learned_addr learned my addr 172.21.15.84:0/2668502846 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:08:53.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.801+0000 7f2da0ff9640 10 client.?.objecter ms_handle_connect 0x7f2da4149ae0 2024-03-25T18:08:53.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.801+0000 7f2da0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.803+0000 7f2da0ff9640 10 client.?.objecter ms_handle_connect 0x7f2da40705b0 2024-03-25T18:08:53.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.803+0000 7f2da0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.805+0000 7f2da0ff9640 10 client.?.objecter ms_handle_connect 0x7f2da41402d0 2024-03-25T18:08:53.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.805+0000 7f2da0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.814+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147706232 0 0) 0x7f2da4142320 con 0x7f2da4149ae0 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.815+0000 7f2da0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.815+0000 7f2da0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.815+0000 7f2da0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.815+0000 7f2da0ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:08:53.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.815+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d90004eb0 con 0x7f2da4149ae0 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036331978 0 0) 0x7f2da41db7d0 con 0x7f2da41402d0 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:08:53.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.818+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da4142320 con 0x7f2da41402d0 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3177738066 0 0) 0x7f2d90004eb0 con 0x7f2da4149ae0 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da41db7d0 con 0x7f2da4149ae0 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d94001830 con 0x7f2da4149ae0 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:53.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086711139 0 0) 0x7f2da4147260 con 0x7f2da40705b0 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:08:53.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.827+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d90004eb0 con 0x7f2da40705b0 2024-03-25T18:08:53.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082309843 0 0) 0x7f2da4142320 con 0x7f2da41402d0 2024-03-25T18:08:53.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da4147260 con 0x7f2da41402d0 2024-03-25T18:08:53.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d980018b0 con 0x7f2da41402d0 2024-03-25T18:08:53.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.850+0000 7f2da0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:53.852 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091734179 0 0) 0x7f2d90004eb0 con 0x7f2da40705b0 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da4142320 con 0x7f2da40705b0 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d9c001e60 con 0x7f2da40705b0 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.852+0000 7f2da0ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:53.854 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:53.855 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:53.855 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:53.855 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2587589806 0 0) 0x7f2da41db7d0 con 0x7f2da4149ae0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 >> v1:172.21.15.157:6789/0 conn(0x7f2da41402d0 legacy=0x7f2da4148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 >> v1:172.21.15.84:6789/0 conn(0x7f2da40705b0 legacy=0x7f2da41dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 1 monclient: found mon.c 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da41424b0 con 0x7f2da4149ae0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2da0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 5 monclient: authenticate success, global_id 4394 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 10 monclient: _renew_subs 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:53.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da41dc400 con 0x7f2da4149ae0 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 10 monclient: _renew_subs 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da4149620 con 0x7f2da4149ae0 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.855+0000 7f2daa972640 1 librados: init done 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.856+0000 7f2daa972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.856+0000 7f2daa972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.856+0000 7f2daa972640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:53.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:53.856+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da41dba10 con 0x7f2da4149ae0 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d94001d20 con 0x7f2da4149ae0 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d940035f0 con 0x7f2da4149ae0 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da0ff9640 10 monclient: dump: 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.156+0000 7f2da27fc640 4 set_mon_vals no callback set 2024-03-25T18:08:54.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.157+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2d94003810 con 0x7f2da4149ae0 2024-03-25T18:08:54.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.157+0000 7f2da3fff640 10 monclient: get_auth_request con 0x7f2d9006e0c0 auth_method 0 2024-03-25T18:08:54.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.158+0000 7f2da0ff9640 10 client.4394.objecter ms_handle_connect 0x7f2d9006e0c0 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f2d940544f0 con 0x7f2da4149ae0 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 10 client.4394.objecter ms_dispatch 0x7f2da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d9404fe30 con 0x7f2da4149ae0 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:54.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.163+0000 7f2da0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:54.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.417+0000 7f2daa972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.417+0000 7f2daa972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.417+0000 7f2daa972640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:08:54.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.417+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2da40627b0 con 0x7f2da4149ae0 2024-03-25T18:08:54.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.441+0000 7f2da0ff9640 1 -- 172.21.15.84:0/2668502846 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f2d94051f30 con 0x7f2da4149ae0 2024-03-25T18:08:54.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.441+0000 7f2da0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.441+0000 7f2da0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:54.443 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:08:54.443 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:54.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 10 librados: watch_flush enter 2024-03-25T18:08:54.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 10 librados: watch_flush exit 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 10 client.4394.objecter successfully canceled tick 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 >> v1:172.21.15.157:6800/1801259026 conn(0x7f2d9006e0c0 legacy=0x7f2d90070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 10 monclient: shutdown 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 >> v1:172.21.15.84:6790/0 conn(0x7f2da4149ae0 legacy=0x7f2da41458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 shutdown_connections 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 >> 172.21.15.84:0/2668502846 conn(0x7f2da40b5d10 msgr2=0x7f2da4140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 shutdown_connections 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 -- 172.21.15.84:0/2668502846 wait complete. 2024-03-25T18:08:54.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:08:54.444+0000 7f2daa972640 1 librados: shutdown 2024-03-25T18:08:54.456 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:57.896 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:54.715 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.903 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:54.723 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.928 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:54.748 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.968 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:54.788 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.974 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:54.794 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.975 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:54.759 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:57.982 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:54.766 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:58.012 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:54.831 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:58.014 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:03:54.798 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:01.262 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.375+0000 7f4163f6d640 10 monclient: get_monmap_and_config 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.375+0000 7f4163f6d640 10 monclient: build_initial_monmap 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.375+0000 7f4163f6d640 10 monclient: monmap: 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:01.375541+0000 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:01.375541+0000 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:01.375 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.375+0000 7f4163f6d640 1 Processor -- start 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.376+0000 7f4163f6d640 1 -- start start 2024-03-25T18:09:01.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.376+0000 7f4163f6d640 10 monclient: init 2024-03-25T18:09:01.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.376+0000 7f4163f6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:01.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.376+0000 7f4163f6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:01.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.376+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f415c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:01.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f415c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f415c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c142520 con 0x7f415c0a98f0 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b28a0 con 0x7f415c1430c0 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b2a30 con 0x7f415c13f710 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:01.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.377+0000 7f4163f6d640 10 monclient(hunting): authenticate will time out at 7185.674970s 2024-03-25T18:09:01.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.391+0000 7f416250a640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f415c1430c0 0x7f415c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:43218/0 (socket says 172.21.15.84:43218) 2024-03-25T18:09:01.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.391+0000 7f416250a640 1 -- 172.21.15.84:0/855986557 learned_addr learned my addr 172.21.15.84:0/855986557 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:01.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.414+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450585226 0 0) 0x7f415c0b28a0 con 0x7f415c1430c0 2024-03-25T18:09:01.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.414+0000 7f4160d07640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.414+0000 7f4160d07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.414+0000 7f4160d07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.414+0000 7f4160d07640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:09:01.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.415+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148003520 con 0x7f415c1430c0 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015873852 0 0) 0x7f415c142520 con 0x7f415c0a98f0 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:09:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.417+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0b28a0 con 0x7f415c0a98f0 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430919181 0 0) 0x7f415c0b2a30 con 0x7f415c13f710 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:09:01.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.420+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c142520 con 0x7f415c13f710 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700189826 0 0) 0x7f4148003520 con 0x7f415c1430c0 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 >> v1:172.21.15.84:6790/0 conn(0x7f415c13f710 legacy=0x7f415c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 >> v1:172.21.15.84:6789/0 conn(0x7f415c0a98f0 legacy=0x7f415c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: _finish_hunting 0 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 monclient: found mon.noname-c 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0b44c0 con 0x7f415c1430c0 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: _finish_auth 0 2024-03-25T18:09:01.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: _check_auth_tickets 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4154001a10 con 0x7f415c1430c0 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4160d07640 10 monclient: dump: 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4163f6d640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:09:01.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.431+0000 7f4163f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:01.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.451+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4154001b70 con 0x7f415c1430c0 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.451+0000 7f4160d07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.451+0000 7f4160d07640 1 -- 172.21.15.84:0/855986557 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4154001f80 con 0x7f415c1430c0 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.451+0000 7f4163f6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.451+0000 7f4163f6d640 4 set_mon_vals no callback set 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4160d07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4160d07640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4160d07640 10 monclient: dump: 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:01.452 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:01.453 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4163f6d640 10 monclient: shutdown 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4163f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.452+0000 7f4163f6d640 1 -- 172.21.15.84:0/855986557 >> v1:172.21.15.157:6789/0 conn(0x7f415c1430c0 legacy=0x7f415c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.453+0000 7f4163f6d640 1 -- 172.21.15.84:0/855986557 shutdown_connections 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.453+0000 7f4163f6d640 1 -- 172.21.15.84:0/855986557 >> 172.21.15.84:0/855986557 conn(0x7f415c0b7ea0 msgr2=0x7f415c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.453+0000 7f4163f6d640 1 -- 172.21.15.84:0/855986557 shutdown_connections 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.453+0000 7f4163f6d640 1 -- 172.21.15.84:0/855986557 wait complete. 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 10 monclient: build_initial_monmap 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 10 monclient: monmap: 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:01.456 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 librados: starting msgr at 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 librados: starting objecter 2024-03-25T18:09:01.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 Processor -- start 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 -- start start 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 librados: setting wanted keys 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 1 librados: calling monclient init 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.454+0000 7f4163f6d640 10 monclient: init 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f415c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f415c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f415c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c1d7500 con 0x7f415c13f710 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c1394a0 con 0x7f415c0a98f0 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c138b50 con 0x7f415c1430c0 2024-03-25T18:09:01.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:01.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.455+0000 7f4163f6d640 10 monclient(hunting): authenticate will time out at 7185.753233s 2024-03-25T18:09:01.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.465+0000 7f416250a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f415c1430c0 0x7f415c134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54010/0 (socket says 172.21.15.84:54010) 2024-03-25T18:09:01.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.465+0000 7f416250a640 1 -- 172.21.15.84:0/3523052056 learned_addr learned my addr 172.21.15.84:0/3523052056 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:01.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.469+0000 7f41527fc640 10 client.?.objecter ms_handle_connect 0x7f415c13f710 2024-03-25T18:09:01.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.469+0000 7f41527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.470+0000 7f41527fc640 10 client.?.objecter ms_handle_connect 0x7f415c1430c0 2024-03-25T18:09:01.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.470+0000 7f41527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.475+0000 7f41527fc640 10 client.?.objecter ms_handle_connect 0x7f415c0a98f0 2024-03-25T18:09:01.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.475+0000 7f41527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182120106 0 0) 0x7f415c138b50 con 0x7f415c1430c0 2024-03-25T18:09:01.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:09:01.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.477+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140004eb0 con 0x7f415c1430c0 2024-03-25T18:09:01.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562642265 0 0) 0x7f4140004eb0 con 0x7f415c1430c0 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c138b50 con 0x7f415c1430c0 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4154001ec0 con 0x7f415c1430c0 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 10 monclient(hunting): dump: 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.486 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.485+0000 7f41527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033566467 0 0) 0x7f415c1d7500 con 0x7f415c13f710 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:09:01.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.487+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140004eb0 con 0x7f415c13f710 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679749482 0 0) 0x7f415c1394a0 con 0x7f415c0a98f0 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:09:01.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.491+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c1d7500 con 0x7f415c0a98f0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 207719773 0 0) 0x7f415c138b50 con 0x7f415c1430c0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 >> v1:172.21.15.157:6789/0 conn(0x7f415c0a98f0 legacy=0x7f415c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 >> v1:172.21.15.84:6789/0 conn(0x7f415c13f710 legacy=0x7f415c1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 1 monclient: found mon.c 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c1d8380 con 0x7f415c1430c0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f41527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:09:01.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 10 monclient: _renew_subs 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c139710 con 0x7f415c1430c0 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 10 monclient: _renew_subs 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c1d7750 con 0x7f415c1430c0 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.495+0000 7f4163f6d640 1 librados: init done 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.496+0000 7f4163f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.496+0000 7f4163f6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.496+0000 7f4163f6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:09:01.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.496+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c0627b0 con 0x7f415c1430c0 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4154004e10 con 0x7f415c1430c0 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f41540015d0 con 0x7f415c1430c0 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 10 monclient: dump: 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:01.808 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:01.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f4153fff640 4 set_mon_vals no callback set 2024-03-25T18:09:01.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.808+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f415401af90 con 0x7f415c1430c0 2024-03-25T18:09:01.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.809+0000 7f4161d09640 10 monclient: get_auth_request con 0x7f414006de80 auth_method 0 2024-03-25T18:09:01.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.809+0000 7f41527fc640 10 client.4400.objecter ms_handle_connect 0x7f414006de80 2024-03-25T18:09:01.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f4154054bf0 con 0x7f415c1430c0 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 10 client.4400.objecter ms_dispatch 0x7f415c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4154050530 con 0x7f415c1430c0 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:01.814+0000 7f41527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.049+0000 7f4163f6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.049+0000 7f4163f6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.049+0000 7f4163f6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:09:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.049+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f415c069f40 con 0x7f415c1430c0 2024-03-25T18:09:02.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.087+0000 7f41527fc640 1 -- 172.21.15.84:0/3523052056 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f4154052630 con 0x7f415c1430c0 2024-03-25T18:09:02.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.087+0000 7f41527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:02.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.087+0000 7f41527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:02.087 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:02.088 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 10 librados: watch_flush enter 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 10 librados: watch_flush exit 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 10 client.4400.objecter successfully canceled tick 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 >> v1:172.21.15.157:6800/1801259026 conn(0x7f414006de80 legacy=0x7f4140070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 10 monclient: shutdown 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 >> v1:172.21.15.84:6790/0 conn(0x7f415c1430c0 legacy=0x7f415c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 shutdown_connections 2024-03-25T18:09:02.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 >> 172.21.15.84:0/3523052056 conn(0x7f415c0b7ea0 msgr2=0x7f415c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:02.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 shutdown_connections 2024-03-25T18:09:02.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 -- 172.21.15.84:0/3523052056 wait complete. 2024-03-25T18:09:02.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:02.090+0000 7f4163f6d640 1 librados: shutdown 2024-03-25T18:09:02.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:08.907 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:09.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.021+0000 7f5c326a8640 10 monclient: get_monmap_and_config 2024-03-25T18:09:09.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.021+0000 7f5c326a8640 10 monclient: build_initial_monmap 2024-03-25T18:09:09.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.021+0000 7f5c326a8640 10 monclient: monmap: 2024-03-25T18:09:09.021 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:09.021826+0000 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:09.021826+0000 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:09.022 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.022+0000 7f5c326a8640 1 Processor -- start 2024-03-25T18:09:09.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.022+0000 7f5c326a8640 1 -- start start 2024-03-25T18:09:09.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.022+0000 7f5c326a8640 10 monclient: init 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c142520 con 0x7f5c2c13f710 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0b28a0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0b2a30 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:09.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.023+0000 7f5c326a8640 10 monclient(hunting): authenticate will time out at 7193.321006s 2024-03-25T18:09:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.031+0000 7f5c30c45640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5c2c1430c0 0x7f5c2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:43518/0 (socket says 172.21.15.84:43518) 2024-03-25T18:09:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.031+0000 7f5c30c45640 1 -- 172.21.15.84:0/1114754589 learned_addr learned my addr 172.21.15.84:0/1114754589 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.042+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540213933 0 0) 0x7f5c2c0b28a0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.043+0000 7f5c2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.043+0000 7f5c2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.043+0000 7f5c2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.043+0000 7f5c2affd640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:09:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.043+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c14003520 con 0x7f5c2c1430c0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703771202 0 0) 0x7f5c14003520 con 0x7f5c2c1430c0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 >> v1:172.21.15.84:6790/0 conn(0x7f5c2c0a98f0 legacy=0x7f5c2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 >> v1:172.21.15.84:6789/0 conn(0x7f5c2c13f710 legacy=0x7f5c2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 monclient: found mon.noname-c 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c0b44c0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: _finish_auth 0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5c24001a10 con 0x7f5c2c1430c0 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c2affd640 10 monclient: dump: 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c326a8640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:09:09.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.060+0000 7f5c326a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c24001b70 con 0x7f5c2c1430c0 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 1 -- 172.21.15.84:0/1114754589 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5c24001fc0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c326a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c326a8640 4 set_mon_vals no callback set 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.072+0000 7f5c2affd640 10 monclient: dump: 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.073 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.074 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.074 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:09.074 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:09.074 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:09.074 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 10 monclient: shutdown 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 1 -- 172.21.15.84:0/1114754589 >> v1:172.21.15.157:6789/0 conn(0x7f5c2c1430c0 legacy=0x7f5c2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 1 -- 172.21.15.84:0/1114754589 shutdown_connections 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 1 -- 172.21.15.84:0/1114754589 >> 172.21.15.84:0/1114754589 conn(0x7f5c2c0b7ea0 msgr2=0x7f5c2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 1 -- 172.21.15.84:0/1114754589 shutdown_connections 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.073+0000 7f5c326a8640 1 -- 172.21.15.84:0/1114754589 wait complete. 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.074+0000 7f5c326a8640 10 monclient: build_initial_monmap 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.074+0000 7f5c326a8640 10 monclient: monmap: 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:09.076 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.074+0000 7f5c326a8640 1 librados: starting msgr at 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.074+0000 7f5c326a8640 1 librados: starting objecter 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.074+0000 7f5c326a8640 1 Processor -- start 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 -- start start 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 librados: setting wanted keys 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 librados: calling monclient init 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient: init 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:09.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c1db990 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c13afc0 con 0x7f5c2c13f710 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c13b2a0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:09.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.075+0000 7f5c326a8640 10 monclient(hunting): authenticate will time out at 7193.373531s 2024-03-25T18:09:09.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.085+0000 7f5c2b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5c2c13f710 0x7f5c2c1dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:43560/0 (socket says 172.21.15.84:43560) 2024-03-25T18:09:09.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.085+0000 7f5c2b7fe640 1 -- 172.21.15.84:0/1313637722 learned_addr learned my addr 172.21.15.84:0/1313637722 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:09.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.089+0000 7f5c28ff9640 10 client.?.objecter ms_handle_connect 0x7f5c2c13f710 2024-03-25T18:09:09.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.089+0000 7f5c28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:09.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.092+0000 7f5c28ff9640 10 client.?.objecter ms_handle_connect 0x7f5c2c0a98f0 2024-03-25T18:09:09.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.092+0000 7f5c28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:09.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.093+0000 7f5c28ff9640 10 client.?.objecter ms_handle_connect 0x7f5c2c1430c0 2024-03-25T18:09:09.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.093+0000 7f5c28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112899622 0 0) 0x7f5c2c13afc0 con 0x7f5c2c13f710 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:09:09.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.102+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18004eb0 con 0x7f5c2c13f710 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859092 0 0) 0x7f5c2c1db990 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:09:09.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.107+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c13afc0 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.117+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49747533 0 0) 0x7f5c2c13b2a0 con 0x7f5c2c1430c0 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.118+0000 7f5c28ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.118+0000 7f5c28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.118+0000 7f5c28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.118+0000 7f5c28ff9640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:09:09.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.118+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c1db990 con 0x7f5c2c1430c0 2024-03-25T18:09:09.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3369996402 0 0) 0x7f5c18004eb0 con 0x7f5c2c13f710 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c2c13b2a0 con 0x7f5c2c13f710 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5c20001a30 con 0x7f5c2c13f710 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:09.122 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.121+0000 7f5c28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776736570 0 0) 0x7f5c2c13afc0 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c18004eb0 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5c1c0018b0 con 0x7f5c2c0a98f0 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:09.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.122+0000 7f5c28ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:09.124 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 769492631 0 0) 0x7f5c2c13b2a0 con 0x7f5c2c13f710 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 >> v1:172.21.15.84:6790/0 conn(0x7f5c2c1430c0 legacy=0x7f5c2c1db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 >> v1:172.21.15.84:6789/0 conn(0x7f5c2c0a98f0 legacy=0x7f5c2c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 1 monclient: found mon.b 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c13b580 con 0x7f5c2c13f710 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 10 monclient: _renew_subs 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c139930 con 0x7f5c2c13f710 2024-03-25T18:09:09.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 10 monclient: _renew_subs 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c13b9a0 con 0x7f5c2c13f710 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.137+0000 7f5c326a8640 1 librados: init done 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.138+0000 7f5c326a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.138+0000 7f5c326a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.138+0000 7f5c326a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.138+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c0627b0 con 0x7f5c2c13f710 2024-03-25T18:09:09.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c20001c70 con 0x7f5c2c13f710 2024-03-25T18:09:09.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:09.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5c20003a60 con 0x7f5c2c13f710 2024-03-25T18:09:09.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c28ff9640 10 monclient: dump: 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.426+0000 7f5c2a7fc640 4 set_mon_vals no callback set 2024-03-25T18:09:09.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.427+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f5c20003c40 con 0x7f5c2c13f710 2024-03-25T18:09:09.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.427+0000 7f5c2bfff640 10 monclient: get_auth_request con 0x7f5c1806df70 auth_method 0 2024-03-25T18:09:09.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.427+0000 7f5c28ff9640 10 client.4405.objecter ms_handle_connect 0x7f5c1806df70 2024-03-25T18:09:09.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.433+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f5c20054900 con 0x7f5c2c13f710 2024-03-25T18:09:09.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.433+0000 7f5c28ff9640 10 client.4405.objecter ms_dispatch 0x7f5c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:09.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.433+0000 7f5c28ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:09.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.433+0000 7f5c28ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:09.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.433+0000 7f5c28ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:09:09.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.435+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c20050410 con 0x7f5c2c13f710 2024-03-25T18:09:09.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.435+0000 7f5c28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:09.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.435+0000 7f5c28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:09.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.664+0000 7f5c326a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.664+0000 7f5c326a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.664+0000 7f5c326a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:09.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.664+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c2c069f40 con 0x7f5c2c13f710 2024-03-25T18:09:09.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.693+0000 7f5c28ff9640 1 -- 172.21.15.84:0/1313637722 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f5c20052340 con 0x7f5c2c13f710 2024-03-25T18:09:09.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.693+0000 7f5c28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.693+0000 7f5c28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:09.694 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:09.694 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:09.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.696+0000 7f5c326a8640 10 librados: watch_flush enter 2024-03-25T18:09:09.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.696+0000 7f5c326a8640 10 librados: watch_flush exit 2024-03-25T18:09:09.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.696+0000 7f5c326a8640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:09:09.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.696+0000 7f5c326a8640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:09:09.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 >> v1:172.21.15.157:6800/1801259026 conn(0x7f5c1806df70 legacy=0x7f5c18070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 10 monclient: shutdown 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 >> v1:172.21.15.157:6789/0 conn(0x7f5c2c13f710 legacy=0x7f5c2c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 shutdown_connections 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 >> 172.21.15.84:0/1313637722 conn(0x7f5c2c0b7ea0 msgr2=0x7f5c2c13e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 shutdown_connections 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 -- 172.21.15.84:0/1313637722 wait complete. 2024-03-25T18:09:09.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:09.697+0000 7f5c326a8640 1 librados: shutdown 2024-03-25T18:09:09.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:16.516 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:16.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.633+0000 7f702c28f640 10 monclient: get_monmap_and_config 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.633+0000 7f702c28f640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.633+0000 7f702c28f640 10 monclient: monmap: 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:16.633962+0000 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:16.633962+0000 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.634 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:16.635 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:16.635 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:16.635 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:16.635 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.634+0000 7f702c28f640 1 Processor -- start 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.634+0000 7f702c28f640 1 -- start start 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.634+0000 7f702c28f640 10 monclient: init 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.634+0000 7f702c28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-a con 0x7f70240a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-c con 0x7f702413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-b con 0x7f70241430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7024142520 con 0x7f70240a98f0 2024-03-25T18:09:16.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240b28a0 con 0x7f702413f710 2024-03-25T18:09:16.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240b2a30 con 0x7f70241430c0 2024-03-25T18:09:16.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.635+0000 7f702c28f640 10 monclient(hunting): authenticate will time out at 7200.933001s 2024-03-25T18:09:16.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.653+0000 7f702a02b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f70240a98f0 0x7f70240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:53820/0 (socket says 172.21.15.84:53820) 2024-03-25T18:09:16.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.653+0000 7f702a02b640 1 -- 172.21.15.84:0/1605417213 learned_addr learned my addr 172.21.15.84:0/1605417213 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.666+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559730248 0 0) 0x7f7024142520 con 0x7f70240a98f0 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7010003520 con 0x7f70240a98f0 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555614684 0 0) 0x7f70240b2a30 con 0x7f70241430c0 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:09:16.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.667+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7024142520 con 0x7f70241430c0 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.676+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642549077 0 0) 0x7f7024142520 con 0x7f70241430c0 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 >> v1:172.21.15.157:6789/0 conn(0x7f702413f710 legacy=0x7f70241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 >> v1:172.21.15.84:6789/0 conn(0x7f70240a98f0 legacy=0x7f70240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 1 monclient: found mon.noname-b 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:09:16.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240b44c0 con 0x7f70241430c0 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f701c0019f0 con 0x7f70241430c0 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f7029029640 10 monclient: dump: 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.678 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f702c28f640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:09:16.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.677+0000 7f702c28f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f701c001b50 con 0x7f70241430c0 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 1 -- 172.21.15.84:0/1605417213 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f701c001fa0 con 0x7f70241430c0 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f7029029640 10 monclient: dump: 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:16.711 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f702c28f640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.707+0000 7f702c28f640 4 set_mon_vals no callback set 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 10 monclient: shutdown 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 1 -- 172.21.15.84:0/1605417213 >> v1:172.21.15.84:6790/0 conn(0x7f70241430c0 legacy=0x7f702414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 1 -- 172.21.15.84:0/1605417213 shutdown_connections 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 1 -- 172.21.15.84:0/1605417213 >> 172.21.15.84:0/1605417213 conn(0x7f70240b7ea0 msgr2=0x7f70240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:16.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 1 -- 172.21.15.84:0/1605417213 shutdown_connections 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.708+0000 7f702c28f640 1 -- 172.21.15.84:0/1605417213 wait complete. 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.709+0000 7f702c28f640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.709+0000 7f702c28f640 10 monclient: monmap: 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:16.713 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.709+0000 7f702c28f640 1 librados: starting msgr at 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.709+0000 7f702c28f640 1 librados: starting objecter 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 Processor -- start 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 -- start start 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 librados: setting wanted keys 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 librados: calling monclient init 2024-03-25T18:09:16.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient: init 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-b con 0x7f70240a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-c con 0x7f702413f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient(hunting): picked mon.noname-a con 0x7f70241430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702414ded0 con 0x7f70241430c0 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702413a1e0 con 0x7f70240a98f0 2024-03-25T18:09:16.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7024135730 con 0x7f702413f710 2024-03-25T18:09:16.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.710+0000 7f702c28f640 10 monclient(hunting): authenticate will time out at 7201.009874s 2024-03-25T18:09:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.729+0000 7f702a82c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f70241430c0 0x7f70241db890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:53866/0 (socket says 172.21.15.84:53866) 2024-03-25T18:09:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.729+0000 7f702a82c640 1 -- 172.21.15.84:0/574131086 learned_addr learned my addr 172.21.15.84:0/574131086 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.730+0000 7f701affd640 10 client.?.objecter ms_handle_connect 0x7f70241430c0 2024-03-25T18:09:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.730+0000 7f701affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.734+0000 7f701affd640 10 client.?.objecter ms_handle_connect 0x7f70240a98f0 2024-03-25T18:09:16.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.734+0000 7f701affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.741+0000 7f701affd640 10 client.?.objecter ms_handle_connect 0x7f702413f710 2024-03-25T18:09:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.741+0000 7f701affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057872227 0 0) 0x7f702414ded0 con 0x7f70241430c0 2024-03-25T18:09:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:09:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.744+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7008004eb0 con 0x7f70241430c0 2024-03-25T18:09:16.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880453513 0 0) 0x7f7024135730 con 0x7f702413f710 2024-03-25T18:09:16.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:09:16.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.754+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f702414ded0 con 0x7f702413f710 2024-03-25T18:09:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594621998 0 0) 0x7f702413a1e0 con 0x7f70240a98f0 2024-03-25T18:09:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:09:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.758+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7024135730 con 0x7f70240a98f0 2024-03-25T18:09:16.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1758036179 0 0) 0x7f7008004eb0 con 0x7f70241430c0 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f702413a1e0 con 0x7f70241430c0 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f701c001f60 con 0x7f70241430c0 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 10 monclient(hunting): dump: 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:16.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:16.771 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:16.771 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:16.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.768+0000 7f701affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:16.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2553915562 0 0) 0x7f702413a1e0 con 0x7f70241430c0 2024-03-25T18:09:16.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 >> v1:172.21.15.84:6790/0 conn(0x7f702413f710 legacy=0x7f7024135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 >> v1:172.21.15.157:6789/0 conn(0x7f70240a98f0 legacy=0x7f702414d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 1 monclient: found mon.a 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70241358c0 con 0x7f70241430c0 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f701affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:09:16.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 10 monclient: _renew_subs 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70240b6e10 con 0x7f70241430c0 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 10 monclient: _renew_subs 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70241dc180 con 0x7f70241430c0 2024-03-25T18:09:16.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.775+0000 7f702c28f640 1 librados: init done 2024-03-25T18:09:16.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.776+0000 7f702c28f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.776+0000 7f702c28f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.776+0000 7f702c28f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:16.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:16.776+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70240b71b0 con 0x7f70241430c0 2024-03-25T18:09:17.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f701c003c20 con 0x7f70241430c0 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f701c003ff0 con 0x7f70241430c0 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f701affd640 10 monclient: dump: 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:17.084 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.082+0000 7f7028828640 4 set_mon_vals no callback set 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.083+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f701c004210 con 0x7f70241430c0 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.083+0000 7f702a02b640 10 monclient: get_auth_request con 0x7f700806ddf0 auth_method 0 2024-03-25T18:09:17.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.084+0000 7f701affd640 10 client.4407.objecter ms_handle_connect 0x7f700806ddf0 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f701c054bf0 con 0x7f70241430c0 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 10 client.4407.objecter ms_dispatch 0x7f7024000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f701c050700 con 0x7f70241430c0 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:17.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.090+0000 7f701affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:17.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.313+0000 7f702c28f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.313+0000 7f702c28f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.313+0000 7f702c28f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:17.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.313+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7024135a50 con 0x7f70241430c0 2024-03-25T18:09:17.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.348+0000 7f701affd640 1 -- 172.21.15.84:0/574131086 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f701c052630 con 0x7f70241430c0 2024-03-25T18:09:17.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.348+0000 7f701affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.348+0000 7f701affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:17.350 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:17.350 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:17.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.350+0000 7f702c28f640 10 librados: watch_flush enter 2024-03-25T18:09:17.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.350+0000 7f702c28f640 10 librados: watch_flush exit 2024-03-25T18:09:17.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.350+0000 7f702c28f640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T18:09:17.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.350+0000 7f702c28f640 10 client.4407.objecter successfully canceled tick 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 >> v1:172.21.15.157:6800/1801259026 conn(0x7f700806ddf0 legacy=0x7f70080702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 10 monclient: shutdown 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 >> v1:172.21.15.84:6789/0 conn(0x7f70241430c0 legacy=0x7f70241db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 shutdown_connections 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 >> 172.21.15.84:0/574131086 conn(0x7f70240b7ea0 msgr2=0x7f7024074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 shutdown_connections 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 -- 172.21.15.84:0/574131086 wait complete. 2024-03-25T18:09:17.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:17.351+0000 7f702c28f640 1 librados: shutdown 2024-03-25T18:09:17.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:24.170 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.283+0000 7f6a34372640 10 monclient: get_monmap_and_config 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.283+0000 7f6a34372640 10 monclient: build_initial_monmap 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.283+0000 7f6a34372640 10 monclient: monmap: 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:24.285288+0000 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:24.285288+0000 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:24.285 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.284+0000 7f6a34372640 1 Processor -- start 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.284+0000 7f6a34372640 1 -- start start 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.284+0000 7f6a34372640 10 monclient: init 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c142520 con 0x7f6a2c13f710 2024-03-25T18:09:24.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0b28a0 con 0x7f6a2c1430c0 2024-03-25T18:09:24.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0b2a30 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:24.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.285+0000 7f6a34372640 10 monclient(hunting): authenticate will time out at 7208.584306s 2024-03-25T18:09:24.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.295+0000 7f6a3290f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6a2c1430c0 0x7f6a2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50942/0 (socket says 172.21.15.84:50942) 2024-03-25T18:09:24.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.295+0000 7f6a3290f640 1 -- 172.21.15.84:0/1515933390 learned_addr learned my addr 172.21.15.84:0/1515933390 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:24.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056479929 0 0) 0x7f6a2c142520 con 0x7f6a2c13f710 2024-03-25T18:09:24.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:09:24.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.303+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a18003520 con 0x7f6a2c13f710 2024-03-25T18:09:24.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433187594 0 0) 0x7f6a2c0b28a0 con 0x7f6a2c1430c0 2024-03-25T18:09:24.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:09:24.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.312+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c142520 con 0x7f6a2c1430c0 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877610375 0 0) 0x7f6a2c0b2a30 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:09:24.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.319+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c0b28a0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250059619 0 0) 0x7f6a2c0b28a0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 >> v1:172.21.15.157:6789/0 conn(0x7f6a2c1430c0 legacy=0x7f6a2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 >> v1:172.21.15.84:6789/0 conn(0x7f6a2c13f710 legacy=0x7f6a2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: _finish_hunting 0 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 monclient: found mon.noname-b 2024-03-25T18:09:24.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0b44c0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: _finish_auth 0 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: _check_auth_tickets 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6a280015b0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a34372640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a34372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a3110c640 10 monclient: dump: 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.328 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.326+0000 7f6a34372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:24.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.341+0000 7f6a3110c640 1 -- 172.21.15.84:0/1515933390 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a28001710 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.341+0000 7f6a3110c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:24.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.341+0000 7f6a34372640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:24.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.341+0000 7f6a34372640 4 set_mon_vals no callback set 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 10 monclient: shutdown 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 1 -- 172.21.15.84:0/1515933390 >> v1:172.21.15.84:6790/0 conn(0x7f6a2c0a98f0 legacy=0x7f6a2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 1 -- 172.21.15.84:0/1515933390 shutdown_connections 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 1 -- 172.21.15.84:0/1515933390 >> 172.21.15.84:0/1515933390 conn(0x7f6a2c0b7ea0 msgr2=0x7f6a2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 1 -- 172.21.15.84:0/1515933390 shutdown_connections 2024-03-25T18:09:24.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.342+0000 7f6a34372640 1 -- 172.21.15.84:0/1515933390 wait complete. 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.343+0000 7f6a34372640 10 monclient: build_initial_monmap 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.343+0000 7f6a34372640 10 monclient: monmap: 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:24.346 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.343+0000 7f6a34372640 1 librados: starting msgr at 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.343+0000 7f6a34372640 1 librados: starting objecter 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 Processor -- start 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 -- start start 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 librados: setting wanted keys 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 librados: calling monclient init 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient: init 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0b0e60 con 0x7f6a2c1430c0 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0aee80 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c1db5b0 con 0x7f6a2c13f710 2024-03-25T18:09:24.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:24.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.344+0000 7f6a34372640 10 monclient(hunting): authenticate will time out at 7208.643601s 2024-03-25T18:09:24.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.349+0000 7f6a3210e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6a2c0a98f0 0x7f6a2c0ae770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51006/0 (socket says 172.21.15.84:51006) 2024-03-25T18:09:24.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.349+0000 7f6a3210e640 1 -- 172.21.15.84:0/522008462 learned_addr learned my addr 172.21.15.84:0/522008462 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:24.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.349+0000 7f6a22ffd640 10 client.?.objecter ms_handle_connect 0x7f6a2c0a98f0 2024-03-25T18:09:24.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.349+0000 7f6a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.353+0000 7f6a22ffd640 10 client.?.objecter ms_handle_connect 0x7f6a2c1430c0 2024-03-25T18:09:24.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.353+0000 7f6a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403251913 0 0) 0x7f6a2c0aee80 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:09:24.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.358+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a10004eb0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.366+0000 7f6a22ffd640 10 client.?.objecter ms_handle_connect 0x7f6a2c13f710 2024-03-25T18:09:24.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.366+0000 7f6a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827386309 0 0) 0x7f6a2c0b0e60 con 0x7f6a2c1430c0 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:09:24.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.368+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c0aee80 con 0x7f6a2c1430c0 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.374+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545949642 0 0) 0x7f6a10004eb0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.374+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a2c0b0e60 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.374+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6a28001d00 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.374+0000 7f6a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.375+0000 7f6a22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.375+0000 7f6a22ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:24.376 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:24.377 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:24.377 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:24.377 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.375+0000 7f6a22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383192643 0 0) 0x7f6a2c1db5b0 con 0x7f6a2c13f710 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:09:24.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.383+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a10004eb0 con 0x7f6a2c13f710 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539124907 0 0) 0x7f6a2c0aee80 con 0x7f6a2c1430c0 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a2c1db5b0 con 0x7f6a2c1430c0 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6a24001b90 con 0x7f6a2c1430c0 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.392+0000 7f6a22ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.393 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.394 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:24.394 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:24.394 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:24.394 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:24.394 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1165275789 0 0) 0x7f6a2c0b0e60 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 >> v1:172.21.15.84:6790/0 conn(0x7f6a2c13f710 legacy=0x7f6a2c0af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 >> v1:172.21.15.84:6789/0 conn(0x7f6a2c1430c0 legacy=0x7f6a2c1e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 1 monclient: found mon.b 2024-03-25T18:09:24.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c1db740 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a34372640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a34372640 10 monclient: _renew_subs 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.393+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c0afa90 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 monclient: _renew_subs 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c14e370 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 1 librados: init done 2024-03-25T18:09:24.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:24.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.394+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c0b3ef0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a280038a0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6a28003c70 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a22ffd640 10 monclient: dump: 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.675 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.673+0000 7f6a3090b640 4 set_mon_vals no callback set 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.674+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f6a28003e90 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.675+0000 7f6a3190d640 10 monclient: get_auth_request con 0x7f6a1006dfb0 auth_method 0 2024-03-25T18:09:24.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.675+0000 7f6a22ffd640 10 client.4417.objecter ms_handle_connect 0x7f6a1006dfb0 2024-03-25T18:09:24.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.676+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f6a28054960 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.676+0000 7f6a22ffd640 10 client.4417.objecter ms_dispatch 0x7f6a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:24.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.676+0000 7f6a22ffd640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:24.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.676+0000 7f6a22ffd640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:24.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.677+0000 7f6a22ffd640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T18:09:24.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.682+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a28050470 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.682+0000 7f6a22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.682+0000 7f6a22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:24.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.910+0000 7f6a34372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.910+0000 7f6a34372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.910+0000 7f6a34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:24.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.910+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2c1db8d0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.949+0000 7f6a22ffd640 1 -- 172.21.15.84:0/522008462 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f6a280523a0 con 0x7f6a2c0a98f0 2024-03-25T18:09:24.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.949+0000 7f6a22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.949+0000 7f6a22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:24.951 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:24.951 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:24.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.951+0000 7f6a34372640 10 librados: watch_flush enter 2024-03-25T18:09:24.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 10 librados: watch_flush exit 2024-03-25T18:09:24.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T18:09:24.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 10 client.4417.objecter successfully canceled tick 2024-03-25T18:09:24.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 >> v1:172.21.15.157:6800/1801259026 conn(0x7f6a1006dfb0 legacy=0x7f6a10070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 10 monclient: shutdown 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 >> v1:172.21.15.157:6789/0 conn(0x7f6a2c0a98f0 legacy=0x7f6a2c0ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 shutdown_connections 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 >> 172.21.15.84:0/522008462 conn(0x7f6a2c0b7ea0 msgr2=0x7f6a2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 shutdown_connections 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 -- 172.21.15.84:0/522008462 wait complete. 2024-03-25T18:09:24.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:24.952+0000 7f6a34372640 1 librados: shutdown 2024-03-25T18:09:24.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:31.770 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.886+0000 7f8cf2d9c640 10 monclient: get_monmap_and_config 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.886+0000 7f8cf2d9c640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.886+0000 7f8cf2d9c640 10 monclient: monmap: 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:31.887915+0000 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:31.887915+0000 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:31.888 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.887+0000 7f8cf2d9c640 1 Processor -- start 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.887+0000 7f8cf2d9c640 1 -- start start 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.887+0000 7f8cf2d9c640 10 monclient: init 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-a con 0x7f8cec0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-b con 0x7f8cec0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-c con 0x7f8cec14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0af940 con 0x7f8cec0b4d60 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0b0a70 con 0x7f8cec14be90 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0b0d50 con 0x7f8cec0b1730 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.888+0000 7f8cf2d9c640 10 monclient(hunting): authenticate will time out at 7216.187315s 2024-03-25T18:09:31.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.890+0000 7f8cebfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8cec0b1730 0x7f8cec0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53226/0 (socket says 172.21.15.84:53226) 2024-03-25T18:09:31.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.890+0000 7f8cebfff640 1 -- 172.21.15.84:0/1596989012 learned_addr learned my addr 172.21.15.84:0/1596989012 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:31.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246741181 0 0) 0x7f8cec0b0d50 con 0x7f8cec0b1730 2024-03-25T18:09:31.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:09:31.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.899+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cdc003520 con 0x7f8cec0b1730 2024-03-25T18:09:31.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682539689 0 0) 0x7f8cec0b0a70 con 0x7f8cec14be90 2024-03-25T18:09:31.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:09:31.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.904+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cec0b0d50 con 0x7f8cec14be90 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081856033 0 0) 0x7f8cec0af940 con 0x7f8cec0b4d60 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.914+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cec0b0a70 con 0x7f8cec0b4d60 2024-03-25T18:09:31.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285648156 0 0) 0x7f8cec0b0a70 con 0x7f8cec0b4d60 2024-03-25T18:09:31.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 >> v1:172.21.15.84:6790/0 conn(0x7f8cec0b1730 legacy=0x7f8cec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 >> v1:172.21.15.157:6789/0 conn(0x7f8cec14be90 legacy=0x7f8cec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 monclient: found mon.noname-a 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0b1030 con 0x7f8cec0b4d60 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8ce00015b0 con 0x7f8cec0b4d60 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:31.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8ceb7fe640 10 monclient: dump: 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8cf2d9c640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:09:31.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.928+0000 7f8cf2d9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:31.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ce0001710 con 0x7f8cec0b4d60 2024-03-25T18:09:31.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:31.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 1 -- 172.21.15.84:0/1596989012 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8ce0001b60 con 0x7f8cec0b4d60 2024-03-25T18:09:31.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8cf2d9c640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8cf2d9c640 4 set_mon_vals no callback set 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.945+0000 7f8ceb7fe640 10 monclient: dump: 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:31.947 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 10 monclient: shutdown 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/1596989012 >> v1:172.21.15.84:6789/0 conn(0x7f8cec0b4d60 legacy=0x7f8cec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/1596989012 shutdown_connections 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/1596989012 >> 172.21.15.84:0/1596989012 conn(0x7f8cec0617a0 msgr2=0x7f8cec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/1596989012 shutdown_connections 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.946+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/1596989012 wait complete. 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 10 monclient: monmap: 2024-03-25T18:09:31.949 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 librados: starting msgr at 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 librados: starting objecter 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 Processor -- start 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 -- start start 2024-03-25T18:09:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 librados: setting wanted keys 2024-03-25T18:09:31.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 1 librados: calling monclient init 2024-03-25T18:09:31.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.947+0000 7f8cf2d9c640 10 monclient: init 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-c con 0x7f8cec0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-b con 0x7f8cec0b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient(hunting): picked mon.noname-a con 0x7f8cec14be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec137e90 con 0x7f8cec14be90 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec138e60 con 0x7f8cec0b4d60 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec139180 con 0x7f8cec0b1730 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.948+0000 7f8cf2d9c640 10 monclient(hunting): authenticate will time out at 7216.247715s 2024-03-25T18:09:31.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.956+0000 7f8cebfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f8cec0b4d60 0x7f8cec13af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34230/0 (socket says 172.21.15.84:34230) 2024-03-25T18:09:31.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.956+0000 7f8cebfff640 1 -- 172.21.15.84:0/3002471120 learned_addr learned my addr 172.21.15.84:0/3002471120 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.957+0000 7f8ce97fa640 10 client.?.objecter ms_handle_connect 0x7f8cec0b4d60 2024-03-25T18:09:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.957+0000 7f8ce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.958+0000 7f8ce97fa640 10 client.?.objecter ms_handle_connect 0x7f8cec0b1730 2024-03-25T18:09:31.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.958+0000 7f8ce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063312971 0 0) 0x7f8cec138e60 con 0x7f8cec0b4d60 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:09:31.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.962+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0004eb0 con 0x7f8cec0b4d60 2024-03-25T18:09:31.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.970+0000 7f8ce97fa640 10 client.?.objecter ms_handle_connect 0x7f8cec14be90 2024-03-25T18:09:31.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.970+0000 7f8ce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026608685 0 0) 0x7f8cd0004eb0 con 0x7f8cec0b4d60 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cec138e60 con 0x7f8cec0b4d60 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8cd8001950 con 0x7f8cec0b4d60 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 10 monclient(hunting): dump: 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.977 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.975+0000 7f8ce97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313053493 0 0) 0x7f8cec139180 con 0x7f8cec0b1730 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:09:31.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.978+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0004eb0 con 0x7f8cec0b1730 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553892918 0 0) 0x7f8cd0004eb0 con 0x7f8cec0b1730 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cec139180 con 0x7f8cec0b1730 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8ce0001a20 con 0x7f8cec0b1730 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.986+0000 7f8ce97fa640 10 monclient(hunting): dump: 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:31.988 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.989 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:31.989 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:31.989 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:31.989 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:31.989 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1307255474 0 0) 0x7f8cec138e60 con 0x7f8cec0b4d60 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 >> v1:172.21.15.84:6790/0 conn(0x7f8cec0b1730 legacy=0x7f8cec131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 >> v1:172.21.15.84:6789/0 conn(0x7f8cec14be90 legacy=0x7f8cec1e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 1 monclient: found mon.b 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec139310 con 0x7f8cec0b4d60 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8ce97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 10 monclient: _renew_subs 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec138470 con 0x7f8cec0b4d60 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 10 monclient: _renew_subs 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec145360 con 0x7f8cec0b4d60 2024-03-25T18:09:31.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.991+0000 7f8cf2d9c640 1 librados: init done 2024-03-25T18:09:31.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.992+0000 7f8cf2d9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.992+0000 7f8cf2d9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.992+0000 7f8cf2d9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:31.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:31.992+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec1401f0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cd80047e0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8cd8004af0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ceaffd640 4 set_mon_vals no callback set 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.279+0000 7f8ce97fa640 10 monclient: dump: 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:32.281 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:32.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.280+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f8cd801ab40 con 0x7f8cec0b4d60 2024-03-25T18:09:32.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.280+0000 7f8cf0b38640 10 monclient: get_auth_request con 0x7f8cd006def0 auth_method 0 2024-03-25T18:09:32.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.281+0000 7f8ce97fa640 10 client.4423.objecter ms_handle_connect 0x7f8cd006def0 2024-03-25T18:09:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.286+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f8cd8054870 con 0x7f8cec0b4d60 2024-03-25T18:09:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.286+0000 7f8ce97fa640 10 client.4423.objecter ms_dispatch 0x7f8cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.286+0000 7f8ce97fa640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:32.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.286+0000 7f8ce97fa640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:32.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.286+0000 7f8ce97fa640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T18:09:32.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.287+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cd80501b0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.287+0000 7f8ce97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:32.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.287+0000 7f8ce97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:32.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.545+0000 7f8cf2d9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.545+0000 7f8cf2d9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.545+0000 7f8cf2d9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:32.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.545+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cec0627b0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.574+0000 7f8ce97fa640 1 -- 172.21.15.84:0/3002471120 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f8cd80522b0 con 0x7f8cec0b4d60 2024-03-25T18:09:32.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.574+0000 7f8ce97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.574+0000 7f8ce97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:32.575 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:32.576 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:32.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 10 librados: watch_flush enter 2024-03-25T18:09:32.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 10 librados: watch_flush exit 2024-03-25T18:09:32.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T18:09:32.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 10 client.4423.objecter successfully canceled tick 2024-03-25T18:09:32.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 >> v1:172.21.15.157:6800/1801259026 conn(0x7f8cd006def0 legacy=0x7f8cd00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 10 monclient: shutdown 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.576+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 >> v1:172.21.15.157:6789/0 conn(0x7f8cec0b4d60 legacy=0x7f8cec13af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.577+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 shutdown_connections 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.577+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 >> 172.21.15.84:0/3002471120 conn(0x7f8cec0617a0 msgr2=0x7f8cec13c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.577+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 shutdown_connections 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.577+0000 7f8cf2d9c640 1 -- 172.21.15.84:0/3002471120 wait complete. 2024-03-25T18:09:32.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:32.577+0000 7f8cf2d9c640 1 librados: shutdown 2024-03-25T18:09:32.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:38.033 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:34.853 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.039 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:34.858 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.071 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:34.890 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.085 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:34.905 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.095 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:34.879 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.099 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:34.918 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.105 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:34.889 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.126 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:34.946 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.144 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:34.928 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:39.395 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.508+0000 7f5ed7997640 10 monclient: get_monmap_and_config 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.508+0000 7f5ed7997640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.508+0000 7f5ed7997640 10 monclient: monmap: 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:39.509834+0000 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:39.509834+0000 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.509+0000 7f5ed7997640 1 Processor -- start 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.509+0000 7f5ed7997640 1 -- start start 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.509+0000 7f5ed7997640 10 monclient: init 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed00b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00af940 con 0x7f5ed00b1730 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00b0a70 con 0x7f5ed014be90 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00b0d50 con 0x7f5ed00b4d60 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.510+0000 7f5ed7997640 10 monclient(hunting): authenticate will time out at 7223.809186s 2024-03-25T18:09:39.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.514+0000 7f5ed5733640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5ed00b4d60 0x7f5ed013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58114/0 (socket says 172.21.15.84:58114) 2024-03-25T18:09:39.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.514+0000 7f5ed5733640 1 -- 172.21.15.84:0/2560619987 learned_addr learned my addr 172.21.15.84:0/2560619987 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439636578 0 0) 0x7f5ed00b0d50 con 0x7f5ed00b4d60 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:09:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.534+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc003520 con 0x7f5ed00b4d60 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143940734 0 0) 0x7f5ed00af940 con 0x7f5ed00b1730 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:09:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.541+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed00b0d50 con 0x7f5ed00b1730 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.546+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326869173 0 0) 0x7f5ebc003520 con 0x7f5ed00b4d60 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 >> v1:172.21.15.157:6789/0 conn(0x7f5ed014be90 legacy=0x7f5ed014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 >> v1:172.21.15.84:6789/0 conn(0x7f5ed00b1730 legacy=0x7f5ed00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 1 monclient: found mon.noname-b 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00b1030 con 0x7f5ed00b4d60 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5ecc001690 con 0x7f5ed00b4d60 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ec7fff640 10 monclient: dump: 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ed7997640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:09:39.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.547+0000 7f5ed7997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ecc0017f0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 1 -- 172.21.15.84:0/2560619987 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5ecc001c00 con 0x7f5ed00b4d60 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ed7997640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:39.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ec7fff640 10 monclient: dump: 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.574+0000 7f5ed7997640 4 set_mon_vals no callback set 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 10 monclient: shutdown 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 1 -- 172.21.15.84:0/2560619987 >> v1:172.21.15.84:6790/0 conn(0x7f5ed00b4d60 legacy=0x7f5ed013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 1 -- 172.21.15.84:0/2560619987 shutdown_connections 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 1 -- 172.21.15.84:0/2560619987 >> 172.21.15.84:0/2560619987 conn(0x7f5ed00617a0 msgr2=0x7f5ed0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 1 -- 172.21.15.84:0/2560619987 shutdown_connections 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.575+0000 7f5ed7997640 1 -- 172.21.15.84:0/2560619987 wait complete. 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.576+0000 7f5ed7997640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.576+0000 7f5ed7997640 10 monclient: monmap: 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:39.578 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.576+0000 7f5ed7997640 1 librados: starting msgr at 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.576+0000 7f5ed7997640 1 librados: starting objecter 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.577+0000 7f5ed7997640 1 Processor -- start 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.577+0000 7f5ed7997640 1 -- start start 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.577+0000 7f5ed7997640 1 librados: setting wanted keys 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.577+0000 7f5ed7997640 1 librados: calling monclient init 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.577+0000 7f5ed7997640 10 monclient: init 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:39.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed00b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed01d7980 con 0x7f5ed00b1730 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed01420a0 con 0x7f5ed014be90 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed01432c0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.578+0000 7f5ed7997640 10 monclient(hunting): authenticate will time out at 7223.877230s 2024-03-25T18:09:39.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.592+0000 7f5ed5733640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5ed00b1730 0x7f5ed0141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:40376/0 (socket says 172.21.15.84:40376) 2024-03-25T18:09:39.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.592+0000 7f5ed5733640 1 -- 172.21.15.84:0/14902550 learned_addr learned my addr 172.21.15.84:0/14902550 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:39.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.593+0000 7f5ec5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ed00b1730 2024-03-25T18:09:39.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.593+0000 7f5ec5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.594+0000 7f5ec5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ed014be90 2024-03-25T18:09:39.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.594+0000 7f5ec5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.596+0000 7f5ec5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ed00b4d60 2024-03-25T18:09:39.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.596+0000 7f5ec5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052971314 0 0) 0x7f5ed01420a0 con 0x7f5ed014be90 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:09:39.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.603+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed014be90 2024-03-25T18:09:39.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182969517 0 0) 0x7f5ed01432c0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:09:39.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.611+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed01420a0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146773226 0 0) 0x7f5ed01d7980 con 0x7f5ed00b1730 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:09:39.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.615+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed01432c0 con 0x7f5ed00b1730 2024-03-25T18:09:39.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2489039780 0 0) 0x7f5eb4004eb0 con 0x7f5ed014be90 2024-03-25T18:09:39.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed01d7980 con 0x7f5ed014be90 2024-03-25T18:09:39.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5ec8001b20 con 0x7f5ed014be90 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.621+0000 7f5ec5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:39.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1320529657 0 0) 0x7f5ed01420a0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed00b4d60 2024-03-25T18:09:39.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5ec0001930 con 0x7f5ed00b4d60 2024-03-25T18:09:39.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.636+0000 7f5ec5ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:39.638 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910191368 0 0) 0x7f5ed01d7980 con 0x7f5ed014be90 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 >> v1:172.21.15.84:6790/0 conn(0x7f5ed00b4d60 legacy=0x7f5ed0138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 >> v1:172.21.15.84:6789/0 conn(0x7f5ed00b1730 legacy=0x7f5ed0141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 1 monclient: found mon.b 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0143940 con 0x7f5ed014be90 2024-03-25T18:09:39.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ec5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ed7997640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ed7997640 10 monclient: _renew_subs 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ed7997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.649+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed0138a70 con 0x7f5ed014be90 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.650+0000 7f5ed7997640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.650+0000 7f5ed7997640 10 monclient: _renew_subs 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.650+0000 7f5ed7997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.650+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed01d8700 con 0x7f5ed014be90 2024-03-25T18:09:39.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.650+0000 7f5ed7997640 1 librados: init done 2024-03-25T18:09:39.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.651+0000 7f5ed7997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.651+0000 7f5ed7997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.651+0000 7f5ed7997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:39.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.651+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed0143130 con 0x7f5ed014be90 2024-03-25T18:09:39.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec8002030 con 0x7f5ed014be90 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5ec8003b10 con 0x7f5ed014be90 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec5ffb640 10 monclient: dump: 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:39.937 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:39.938 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:39.938 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:39.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.935+0000 7f5ec77fe640 4 set_mon_vals no callback set 2024-03-25T18:09:39.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.936+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f5ec8019ae0 con 0x7f5ed014be90 2024-03-25T18:09:39.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.937+0000 7f5ed5733640 10 monclient: get_auth_request con 0x7f5eb406df20 auth_method 0 2024-03-25T18:09:39.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.937+0000 7f5ec5ffb640 10 client.4429.objecter ms_handle_connect 0x7f5eb406df20 2024-03-25T18:09:39.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f5ec8053780 con 0x7f5ed014be90 2024-03-25T18:09:39.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 10 client.4429.objecter ms_dispatch 0x7f5ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:39.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:39.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:39.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T18:09:39.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ec804f0c0 con 0x7f5ed014be90 2024-03-25T18:09:39.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:39.943+0000 7f5ec5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:40.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.186+0000 7f5ed7997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.186+0000 7f5ed7997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.186+0000 7f5ed7997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:40.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.186+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed01401f0 con 0x7f5ed014be90 2024-03-25T18:09:40.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.251+0000 7f5ec5ffb640 1 -- 172.21.15.84:0/14902550 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f5ec80511c0 con 0x7f5ed014be90 2024-03-25T18:09:40.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.251+0000 7f5ec5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.251+0000 7f5ec5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:40.252 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:40.252 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:40.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.253+0000 7f5ed7997640 10 librados: watch_flush enter 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 10 librados: watch_flush exit 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 10 client.4429.objecter successfully canceled tick 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 >> v1:172.21.15.157:6800/1801259026 conn(0x7f5eb406df20 legacy=0x7f5eb40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 10 monclient: shutdown 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 >> v1:172.21.15.157:6789/0 conn(0x7f5ed014be90 legacy=0x7f5ed01d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 shutdown_connections 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 >> 172.21.15.84:0/14902550 conn(0x7f5ed00617a0 msgr2=0x7f5ed0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 shutdown_connections 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 -- 172.21.15.84:0/14902550 wait complete. 2024-03-25T18:09:40.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:40.254+0000 7f5ed7997640 1 librados: shutdown 2024-03-25T18:09:40.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:47.074 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:47.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.189+0000 7f4d69e7b640 10 monclient: get_monmap_and_config 2024-03-25T18:09:47.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.189+0000 7f4d69e7b640 10 monclient: build_initial_monmap 2024-03-25T18:09:47.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.189+0000 7f4d69e7b640 10 monclient: monmap: 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:47.190860+0000 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:47.190860+0000 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:47.191 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.190+0000 7f4d69e7b640 1 Processor -- start 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.190+0000 7f4d69e7b640 1 -- start start 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.190+0000 7f4d69e7b640 10 monclient: init 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6406ab80 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6406af60 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d640b00b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640b0500 con 0x7f4d6406ab80 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6413ffa0 con 0x7f4d640b00b0 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640af520 con 0x7f4d6406af60 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:47.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.191+0000 7f4d69e7b640 10 monclient(hunting): authenticate will time out at 7231.490272s 2024-03-25T18:09:47.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.197+0000 7f4d63fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4d640b00b0 0x7f4d6414dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:48036/0 (socket says 172.21.15.84:48036) 2024-03-25T18:09:47.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.197+0000 7f4d63fff640 1 -- 172.21.15.84:0/3400098922 learned_addr learned my addr 172.21.15.84:0/3400098922 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347618287 0 0) 0x7f4d6413ffa0 con 0x7f4d640b00b0 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:09:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.210+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d4c003520 con 0x7f4d640b00b0 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760495899 0 0) 0x7f4d640af520 con 0x7f4d6406af60 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:09:47.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.214+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6413ffa0 con 0x7f4d6406af60 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2804380635 0 0) 0x7f4d4c003520 con 0x7f4d640b00b0 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 >> v1:172.21.15.84:6790/0 conn(0x7f4d6406af60 legacy=0x7f4d6413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 >> v1:172.21.15.84:6789/0 conn(0x7f4d6406ab80 legacy=0x7f4d640ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 monclient: found mon.noname-c 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:47.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d64072890 con 0x7f4d640b00b0 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d5c001a10 con 0x7f4d640b00b0 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d627fc640 10 monclient: dump: 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.229 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.230 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.230 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d69e7b640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:09:47.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.226+0000 7f4d69e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:47.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d5c001b70 con 0x7f4d640b00b0 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 1 -- 172.21.15.84:0/3400098922 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d5c001f80 con 0x7f4d640b00b0 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d69e7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d69e7b640 4 set_mon_vals no callback set 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.239+0000 7f4d627fc640 10 monclient: dump: 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.241 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.242 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.242 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.242 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 10 monclient: shutdown 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 1 -- 172.21.15.84:0/3400098922 >> v1:172.21.15.157:6789/0 conn(0x7f4d640b00b0 legacy=0x7f4d6414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 1 -- 172.21.15.84:0/3400098922 shutdown_connections 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 1 -- 172.21.15.84:0/3400098922 >> 172.21.15.84:0/3400098922 conn(0x7f4d640b4a80 msgr2=0x7f4d640b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 1 -- 172.21.15.84:0/3400098922 shutdown_connections 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.240+0000 7f4d69e7b640 1 -- 172.21.15.84:0/3400098922 wait complete. 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 10 monclient: build_initial_monmap 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 10 monclient: monmap: 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:47.244 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 librados: starting msgr at 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 librados: starting objecter 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 Processor -- start 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 -- start start 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 librados: setting wanted keys 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 1 librados: calling monclient init 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.241+0000 7f4d69e7b640 10 monclient: init 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6406ab80 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:47.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d640b00b0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d64135ee0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64137b10 con 0x7f4d6406ab80 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64136860 con 0x7f4d64135ee0 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64137d80 con 0x7f4d640b00b0 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:47.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.242+0000 7f4d69e7b640 10 monclient(hunting): authenticate will time out at 7231.541420s 2024-03-25T18:09:47.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.251+0000 7f4d637fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4d6406ab80 0x7f4d64136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44982/0 (socket says 172.21.15.84:44982) 2024-03-25T18:09:47.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.252+0000 7f4d637fe640 1 -- 172.21.15.84:0/351237913 learned_addr learned my addr 172.21.15.84:0/351237913 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:47.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.254+0000 7f4d4bfff640 10 client.?.objecter ms_handle_connect 0x7f4d6406ab80 2024-03-25T18:09:47.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.254+0000 7f4d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.256+0000 7f4d4bfff640 10 client.?.objecter ms_handle_connect 0x7f4d64135ee0 2024-03-25T18:09:47.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.256+0000 7f4d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137888357 0 0) 0x7f4d64136860 con 0x7f4d64135ee0 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:09:47.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.263+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50004eb0 con 0x7f4d64135ee0 2024-03-25T18:09:47.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.267+0000 7f4d4bfff640 10 client.?.objecter ms_handle_connect 0x7f4d640b00b0 2024-03-25T18:09:47.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.267+0000 7f4d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494330778 0 0) 0x7f4d64137b10 con 0x7f4d6406ab80 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:09:47.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.280+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d64136860 con 0x7f4d6406ab80 2024-03-25T18:09:47.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3727468894 0 0) 0x7f4d50004eb0 con 0x7f4d64135ee0 2024-03-25T18:09:47.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d64137b10 con 0x7f4d64135ee0 2024-03-25T18:09:47.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d5c001e80 con 0x7f4d64135ee0 2024-03-25T18:09:47.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): dump: 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197687952 0 0) 0x7f4d64137d80 con 0x7f4d640b00b0 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:09:47.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.289+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50004eb0 con 0x7f4d640b00b0 2024-03-25T18:09:47.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951586413 0 0) 0x7f4d50004eb0 con 0x7f4d640b00b0 2024-03-25T18:09:47.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d64137d80 con 0x7f4d640b00b0 2024-03-25T18:09:47.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d58001930 con 0x7f4d640b00b0 2024-03-25T18:09:47.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.304+0000 7f4d4bfff640 10 monclient(hunting): dump: 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.306 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158061597 0 0) 0x7f4d64136860 con 0x7f4d6406ab80 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d50004eb0 con 0x7f4d6406ab80 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d540019b0 con 0x7f4d6406ab80 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.311+0000 7f4d4bfff640 10 monclient(hunting): dump: 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.312 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.313 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.313 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.313 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.313 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.313 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3249367448 0 0) 0x7f4d64137b10 con 0x7f4d64135ee0 2024-03-25T18:09:47.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 >> v1:172.21.15.84:6790/0 conn(0x7f4d640b00b0 legacy=0x7f4d641d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 >> v1:172.21.15.84:6789/0 conn(0x7f4d6406ab80 legacy=0x7f4d64136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 1 monclient: found mon.b 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d64138060 con 0x7f4d64135ee0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 10 monclient: _finish_auth 0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 10 monclient: _renew_subs 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d64138340 con 0x7f4d64135ee0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 10 monclient: _renew_subs 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d64145fa0 con 0x7f4d64135ee0 2024-03-25T18:09:47.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.315+0000 7f4d69e7b640 1 librados: init done 2024-03-25T18:09:47.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.316+0000 7f4d69e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.316+0000 7f4d69e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.316+0000 7f4d69e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.316+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d64135860 con 0x7f4d64135ee0 2024-03-25T18:09:47.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d5c002020 con 0x7f4d64135ee0 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4d5c004fb0 con 0x7f4d64135ee0 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d4bfff640 10 monclient: dump: 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:47.595 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.593+0000 7f4d61ffb640 4 set_mon_vals no callback set 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.594+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f4d5c01af80 con 0x7f4d64135ee0 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.595+0000 7f4d637fe640 10 monclient: get_auth_request con 0x7f4d5006e110 auth_method 0 2024-03-25T18:09:47.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.595+0000 7f4d4bfff640 10 client.4435.objecter ms_handle_connect 0x7f4d5006e110 2024-03-25T18:09:47.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f4d5c054ae0 con 0x7f4d64135ee0 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 10 client.4435.objecter ms_dispatch 0x7f4d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d5c0505f0 con 0x7f4d64135ee0 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.598+0000 7f4d4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:47.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.817+0000 7f4d69e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.817+0000 7f4d69e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.817+0000 7f4d69e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:09:47.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.817+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d640adf40 con 0x7f4d64135ee0 2024-03-25T18:09:47.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.839+0000 7f4d4bfff640 1 -- 172.21.15.84:0/351237913 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f4d5c052520 con 0x7f4d64135ee0 2024-03-25T18:09:47.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.839+0000 7f4d4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.839+0000 7f4d4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:47.841 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:47.841 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 10 librados: watch_flush enter 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 10 librados: watch_flush exit 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 >> v1:172.21.15.157:6800/1801259026 conn(0x7f4d5006e110 legacy=0x7f4d500705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 10 monclient: shutdown 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 >> v1:172.21.15.157:6789/0 conn(0x7f4d64135ee0 legacy=0x7f4d641456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 shutdown_connections 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.842+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 >> 172.21.15.84:0/351237913 conn(0x7f4d640b4a80 msgr2=0x7f4d6413f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.843+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 shutdown_connections 2024-03-25T18:09:47.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.843+0000 7f4d69e7b640 1 -- 172.21.15.84:0/351237913 wait complete. 2024-03-25T18:09:47.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:47.843+0000 7f4d69e7b640 1 librados: shutdown 2024-03-25T18:09:47.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:54.663 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:54.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.777+0000 7ff545b96640 10 monclient: get_monmap_and_config 2024-03-25T18:09:54.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.777+0000 7ff545b96640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.777+0000 7ff545b96640 10 monclient: monmap: 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:09:54.777942+0000 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:09:54.777942+0000 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:09:54.778 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.778+0000 7ff545b96640 1 Processor -- start 2024-03-25T18:09:54.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.778+0000 7ff545b96640 1 -- start start 2024-03-25T18:09:54.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.778+0000 7ff545b96640 10 monclient: init 2024-03-25T18:09:54.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-a con 0x7ff5400705b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-b con 0x7ff5401402d0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-c con 0x7ff540149ae0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5400718c0 con 0x7ff5400705b0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5400714c0 con 0x7ff540149ae0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff540072b50 con 0x7ff5401402d0 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.779+0000 7ff545b96640 10 monclient(hunting): authenticate will time out at 7239.077117s 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.786+0000 7ff53effd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff5401402d0 0x7ff540140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44988/0 (socket says 172.21.15.84:44988) 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.786+0000 7ff53effd640 1 -- 172.21.15.84:0/2882031636 learned_addr learned my addr 172.21.15.84:0/2882031636 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:54.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284165977 0 0) 0x7ff540072b50 con 0x7ff5401402d0 2024-03-25T18:09:54.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.798+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff528003520 con 0x7ff5401402d0 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423857104 0 0) 0x7ff5400714c0 con 0x7ff540149ae0 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:09:54.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.799+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540072b50 con 0x7ff540149ae0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211996430 0 0) 0x7ff540072b50 con 0x7ff540149ae0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 >> v1:172.21.15.84:6790/0 conn(0x7ff5401402d0 legacy=0x7ff540140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 >> v1:172.21.15.84:6789/0 conn(0x7ff5400705b0 legacy=0x7ff54013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54013dd10 con 0x7ff540149ae0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff53e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff545b96640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.804+0000 7ff545b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.805+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff5380019b0 con 0x7ff540149ae0 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.805+0000 7ff53e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.805+0000 7ff53e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.805+0000 7ff53e7fc640 10 monclient: dump: 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.805+0000 7ff545b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.818+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff538001c40 con 0x7ff540149ae0 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff53e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff53e7fc640 1 -- 172.21.15.84:0/2882031636 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff538001f50 con 0x7ff540149ae0 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 4 set_mon_vals no callback set 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff53e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff53e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff53e7fc640 10 monclient: dump: 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.819 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.820 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:54.820 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:54.820 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:54.820 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:54.820 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 10 monclient: shutdown 2024-03-25T18:09:54.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:54.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 1 -- 172.21.15.84:0/2882031636 >> v1:172.21.15.157:6789/0 conn(0x7ff540149ae0 legacy=0x7ff54014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 1 -- 172.21.15.84:0/2882031636 shutdown_connections 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 1 -- 172.21.15.84:0/2882031636 >> 172.21.15.84:0/2882031636 conn(0x7ff5400b5d10 msgr2=0x7ff5400b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 1 -- 172.21.15.84:0/2882031636 shutdown_connections 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.819+0000 7ff545b96640 1 -- 172.21.15.84:0/2882031636 wait complete. 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.820+0000 7ff545b96640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.820+0000 7ff545b96640 10 monclient: monmap: 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:09:54.822 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 librados: starting msgr at 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 librados: starting objecter 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 Processor -- start 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 -- start start 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 librados: setting wanted keys 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 1 librados: calling monclient init 2024-03-25T18:09:54.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.821+0000 7ff545b96640 10 monclient: init 2024-03-25T18:09:54.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:54.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-a con 0x7ff5400705b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:09:54.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-c con 0x7ff5401402d0 addr v1:172.21.15.84:6790/0 2024-03-25T18:09:54.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient(hunting): picked mon.noname-b con 0x7ff540149ae0 addr v1:172.21.15.157:6789/0 2024-03-25T18:09:54.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff540137710 con 0x7ff5400705b0 2024-03-25T18:09:54.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5401379f0 con 0x7ff540149ae0 2024-03-25T18:09:54.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff540143130 con 0x7ff5401402d0 2024-03-25T18:09:54.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.822+0000 7ff545b96640 10 monclient(hunting): authenticate will time out at 7239.120311s 2024-03-25T18:09:54.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.834+0000 7ff53f7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff5400705b0 0x7ff540143650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:36240/0 (socket says 172.21.15.84:36240) 2024-03-25T18:09:54.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.834+0000 7ff53f7fe640 1 -- 172.21.15.84:0/4023605761 learned_addr learned my addr 172.21.15.84:0/4023605761 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.839+0000 7ff527fff640 10 client.?.objecter ms_handle_connect 0x7ff5401402d0 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.839+0000 7ff527fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.839+0000 7ff527fff640 10 client.?.objecter ms_handle_connect 0x7ff5400705b0 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.839+0000 7ff527fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.840+0000 7ff527fff640 10 client.?.objecter ms_handle_connect 0x7ff540149ae0 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.840+0000 7ff527fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325071310 0 0) 0x7ff540137710 con 0x7ff5400705b0 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.862+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff52c004eb0 con 0x7ff5400705b0 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835968187 0 0) 0x7ff540143130 con 0x7ff5401402d0 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:09:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.863+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540137710 con 0x7ff5401402d0 2024-03-25T18:09:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602629549 0 0) 0x7ff5401379f0 con 0x7ff540149ae0 2024-03-25T18:09:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:09:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.869+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540143130 con 0x7ff540149ae0 2024-03-25T18:09:54.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 32509936 0 0) 0x7ff52c004eb0 con 0x7ff5400705b0 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5401379f0 con 0x7ff5400705b0 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff5300019d0 con 0x7ff5400705b0 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 10 monclient(hunting): dump: 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:54.880 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.879+0000 7ff527fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433033642 0 0) 0x7ff540137710 con 0x7ff5401402d0 2024-03-25T18:09:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff52c004eb0 con 0x7ff5401402d0 2024-03-25T18:09:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff534001950 con 0x7ff5401402d0 2024-03-25T18:09:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.887+0000 7ff527fff640 10 monclient(hunting): dump: 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:54.888 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2310157770 0 0) 0x7ff5401379f0 con 0x7ff5400705b0 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 >> v1:172.21.15.84:6790/0 conn(0x7ff5401402d0 legacy=0x7ff540132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 >> v1:172.21.15.157:6789/0 conn(0x7ff540149ae0 legacy=0x7ff540133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 1 monclient: found mon.a 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54013ac30 con 0x7ff5400705b0 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 10 monclient: _renew_subs 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540143d60 con 0x7ff5400705b0 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 10 monclient: _renew_subs 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54013f090 con 0x7ff5400705b0 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.893+0000 7ff545b96640 1 librados: init done 2024-03-25T18:09:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.894+0000 7ff545b96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.894+0000 7ff545b96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.894+0000 7ff545b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:54.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:54.894+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5401432c0 con 0x7ff5400705b0 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff530003610 con 0x7ff5400705b0 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff530003a20 con 0x7ff5400705b0 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 10 monclient: dump: 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:09:55.218 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:09:55.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff53dffb640 4 set_mon_vals no callback set 2024-03-25T18:09:55.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.218+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7ff530003c40 con 0x7ff5400705b0 2024-03-25T18:09:55.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.219+0000 7ff53effd640 10 monclient: get_auth_request con 0x7ff52c06df70 auth_method 0 2024-03-25T18:09:55.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.219+0000 7ff527fff640 10 client.4437.objecter ms_handle_connect 0x7ff52c06df70 2024-03-25T18:09:55.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7ff5300547f0 con 0x7ff5400705b0 2024-03-25T18:09:55.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 10 client.4437.objecter ms_dispatch 0x7ff540000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:55.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:55.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:55.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T18:09:55.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff530050300 con 0x7ff5400705b0 2024-03-25T18:09:55.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:55.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.225+0000 7ff527fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:55.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.479+0000 7ff545b96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.479+0000 7ff545b96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.479+0000 7ff545b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:09:55.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.479+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff540142f30 con 0x7ff5400705b0 2024-03-25T18:09:55.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.515+0000 7ff527fff640 1 -- 172.21.15.84:0/4023605761 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7ff530052230 con 0x7ff5400705b0 2024-03-25T18:09:55.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.515+0000 7ff527fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.515+0000 7ff527fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:55.516 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:09:55.516 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.517+0000 7ff545b96640 10 librados: watch_flush enter 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 10 librados: watch_flush exit 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 10 client.4437.objecter successfully canceled tick 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 >> v1:172.21.15.157:6800/1801259026 conn(0x7ff52c06df70 legacy=0x7ff52c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 10 monclient: shutdown 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 >> v1:172.21.15.84:6789/0 conn(0x7ff5400705b0 legacy=0x7ff540143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 shutdown_connections 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 >> 172.21.15.84:0/4023605761 conn(0x7ff5400b5d10 msgr2=0x7ff54013c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 shutdown_connections 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 -- 172.21.15.84:0/4023605761 wait complete. 2024-03-25T18:09:55.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:09:55.518+0000 7ff545b96640 1 librados: shutdown 2024-03-25T18:09:55.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:58.073 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:54.892 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.081 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:54.900 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.121 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:54.940 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.141 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:54.960 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.150 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:54.970 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.154 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:54.938 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.157 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:54.940 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.180 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:54.999 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.188 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:04:54.972 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:02.336 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.449+0000 7f35ef532640 10 monclient: get_monmap_and_config 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.449+0000 7f35ef532640 10 monclient: build_initial_monmap 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.450+0000 7f35ef532640 10 monclient: monmap: 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:02.450372+0000 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:02.450372+0000 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:02.450 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.450+0000 7f35ef532640 1 Processor -- start 2024-03-25T18:10:02.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 1 -- start start 2024-03-25T18:10:02.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 10 monclient: init 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-c con 0x7f35e80b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.451+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-b con 0x7f35e80b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-a con 0x7f35e814be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:02.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80af940 con 0x7f35e814be90 2024-03-25T18:10:02.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80b0a70 con 0x7f35e80b4d60 2024-03-25T18:10:02.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80b0d50 con 0x7f35e80b1730 2024-03-25T18:10:02.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:02.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.452+0000 7f35ef532640 10 monclient(hunting): authenticate will time out at 7246.749970s 2024-03-25T18:10:02.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.470+0000 7f35ecacd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f35e80b1730 0x7f35e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36172/0 (socket says 172.21.15.84:36172) 2024-03-25T18:10:02.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.470+0000 7f35ecacd640 1 -- 172.21.15.84:0/1109001055 learned_addr learned my addr 172.21.15.84:0/1109001055 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287451713 0 0) 0x7f35e80b0a70 con 0x7f35e80b4d60 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d4003520 con 0x7f35e80b4d60 2024-03-25T18:10:02.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950106686 0 0) 0x7f35e80b0d50 con 0x7f35e80b1730 2024-03-25T18:10:02.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:10:02.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.484+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e80b0a70 con 0x7f35e80b1730 2024-03-25T18:10:02.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335574264 0 0) 0x7f35e80af940 con 0x7f35e814be90 2024-03-25T18:10:02.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:10:02.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.487+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e80b0d50 con 0x7f35e814be90 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987660305 0 0) 0x7f35e80b0a70 con 0x7f35e80b1730 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 >> v1:172.21.15.157:6789/0 conn(0x7f35e80b4d60 legacy=0x7f35e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 >> v1:172.21.15.84:6789/0 conn(0x7f35e814be90 legacy=0x7f35e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 monclient: found mon.noname-b 2024-03-25T18:10:02.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e80b1030 con 0x7f35e80b1730 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35d8001670 con 0x7f35e80b1730 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35dffff640 10 monclient: dump: 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.501 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.502 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.502 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35ef532640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:10:02.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.500+0000 7f35ef532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:02.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d80017d0 con 0x7f35e80b1730 2024-03-25T18:10:02.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 1 -- 172.21.15.84:0/1109001055 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35d8001be0 con 0x7f35e80b1730 2024-03-25T18:10:02.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35ef532640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35ef532640 4 set_mon_vals no callback set 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.511+0000 7f35dffff640 10 monclient: dump: 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.512 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 10 monclient: shutdown 2024-03-25T18:10:02.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:02.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 1 -- 172.21.15.84:0/1109001055 >> v1:172.21.15.84:6790/0 conn(0x7f35e80b1730 legacy=0x7f35e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 1 -- 172.21.15.84:0/1109001055 shutdown_connections 2024-03-25T18:10:02.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 1 -- 172.21.15.84:0/1109001055 >> 172.21.15.84:0/1109001055 conn(0x7f35e80617a0 msgr2=0x7f35e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 1 -- 172.21.15.84:0/1109001055 shutdown_connections 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.512+0000 7f35ef532640 1 -- 172.21.15.84:0/1109001055 wait complete. 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 10 monclient: build_initial_monmap 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 10 monclient: monmap: 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 librados: starting msgr at 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 librados: starting objecter 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 Processor -- start 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 -- start start 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 librados: setting wanted keys 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 1 librados: calling monclient init 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.513+0000 7f35ef532640 10 monclient: init 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-a con 0x7f35e80b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-b con 0x7f35e80b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient(hunting): picked mon.noname-c con 0x7f35e814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8144d80 con 0x7f35e80b1730 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8145020 con 0x7f35e80b4d60 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8137b90 con 0x7f35e814be90 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:02.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.514+0000 7f35ef532640 10 monclient(hunting): authenticate will time out at 7246.812466s 2024-03-25T18:10:02.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.522+0000 7f35ecacd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f35e80b4d60 0x7f35e81dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:47332/0 (socket says 172.21.15.84:47332) 2024-03-25T18:10:02.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.522+0000 7f35ecacd640 1 -- 172.21.15.84:0/3554166138 learned_addr learned my addr 172.21.15.84:0/3554166138 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:02.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.523+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e80b4d60 2024-03-25T18:10:02.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.523+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.528+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e80b1730 2024-03-25T18:10:02.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.528+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.528+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e814be90 2024-03-25T18:10:02.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.528+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650441994 0 0) 0x7f35e8145020 con 0x7f35e80b4d60 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:10:02.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.529+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc004eb0 con 0x7f35e80b4d60 2024-03-25T18:10:02.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230451882 0 0) 0x7f35e8137b90 con 0x7f35e814be90 2024-03-25T18:10:02.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:10:02.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.535+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e8145020 con 0x7f35e814be90 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653241893 0 0) 0x7f35e8144d80 con 0x7f35e80b1730 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:10:02.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.536+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e8137b90 con 0x7f35e80b1730 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 665864348 0 0) 0x7f35cc004eb0 con 0x7f35e80b4d60 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e8144d80 con 0x7f35e80b4d60 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35d8001c20 con 0x7f35e80b4d60 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.539 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.540 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.540 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.540 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.540 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.538+0000 7f35ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:02.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2166791968 0 0) 0x7f35e8137b90 con 0x7f35e80b1730 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35cc004eb0 con 0x7f35e80b1730 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35e40017b0 con 0x7f35e80b1730 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.541+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.542 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.543 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382559885 0 0) 0x7f35e8145020 con 0x7f35e814be90 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e8137b90 con 0x7f35e814be90 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35e0001b50 con 0x7f35e814be90 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.543+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.544 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.545 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 18244429 0 0) 0x7f35cc004eb0 con 0x7f35e80b1730 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 >> v1:172.21.15.84:6790/0 conn(0x7f35e814be90 legacy=0x7f35e81e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 >> v1:172.21.15.157:6789/0 conn(0x7f35e80b4d60 legacy=0x7f35e81dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 1 monclient: found mon.a 2024-03-25T18:10:02.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e81405f0 con 0x7f35e80b1730 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 10 monclient: _renew_subs 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e81408c0 con 0x7f35e80b1730 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 10 monclient: _renew_subs 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e81325e0 con 0x7f35e80b1730 2024-03-25T18:10:02.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.547+0000 7f35ef532640 1 librados: init done 2024-03-25T18:10:02.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.548+0000 7f35ef532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.548+0000 7f35ef532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.548+0000 7f35ef532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:02.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.548+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e81451b0 con 0x7f35e80b1730 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e4001cc0 con 0x7f35e80b1730 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f35e40037b0 con 0x7f35e80b1730 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35ddffb640 10 monclient: dump: 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:02.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:02.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.568+0000 7f35df7fe640 4 set_mon_vals no callback set 2024-03-25T18:10:02.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.569+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f35e4019780 con 0x7f35e80b1730 2024-03-25T18:10:02.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.570+0000 7f35ecacd640 10 monclient: get_auth_request con 0x7f35cc06e110 auth_method 0 2024-03-25T18:10:02.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.570+0000 7f35ddffb640 10 client.4443.objecter ms_handle_connect 0x7f35cc06e110 2024-03-25T18:10:02.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.573+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f35e40534b0 con 0x7f35e80b1730 2024-03-25T18:10:02.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.573+0000 7f35ddffb640 10 client.4443.objecter ms_dispatch 0x7f35e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:02.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.573+0000 7f35ddffb640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:02.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.573+0000 7f35ddffb640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:02.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.573+0000 7f35ddffb640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T18:10:02.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.850+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35e404efc0 con 0x7f35e80b1730 2024-03-25T18:10:02.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.850+0000 7f35ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:02.850+0000 7f35ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:03.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.098+0000 7f35ef532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:03.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.098+0000 7f35ef532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:03.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.098+0000 7f35ef532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:03.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.098+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35e80627b0 con 0x7f35e80b1730 2024-03-25T18:10:03.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.245+0000 7f35ddffb640 1 -- 172.21.15.84:0/3554166138 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f35e4050ef0 con 0x7f35e80b1730 2024-03-25T18:10:03.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.245+0000 7f35ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:03.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.245+0000 7f35ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:03.246 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:03.246 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:03.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.247+0000 7f35ef532640 10 librados: watch_flush enter 2024-03-25T18:10:03.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.247+0000 7f35ef532640 10 librados: watch_flush exit 2024-03-25T18:10:03.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.247+0000 7f35ef532640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T18:10:03.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.247+0000 7f35ef532640 10 client.4443.objecter successfully canceled tick 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 >> v1:172.21.15.157:6800/1801259026 conn(0x7f35cc06e110 legacy=0x7f35cc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 10 monclient: shutdown 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 >> v1:172.21.15.84:6789/0 conn(0x7f35e80b1730 legacy=0x7f35e8131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 shutdown_connections 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 >> 172.21.15.84:0/3554166138 conn(0x7f35e80617a0 msgr2=0x7f35e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 shutdown_connections 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 -- 172.21.15.84:0/3554166138 wait complete. 2024-03-25T18:10:03.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:03.248+0000 7f35ef532640 1 librados: shutdown 2024-03-25T18:10:03.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:10.067 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:10.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.183+0000 7f13372ff640 10 monclient: get_monmap_and_config 2024-03-25T18:10:10.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.183+0000 7f13372ff640 10 monclient: build_initial_monmap 2024-03-25T18:10:10.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.183+0000 7f13372ff640 10 monclient: monmap: 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:10.183827+0000 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:10.183827+0000 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.184+0000 7f13372ff640 1 Processor -- start 2024-03-25T18:10:10.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.184+0000 7f13372ff640 1 -- start start 2024-03-25T18:10:10.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.184+0000 7f13372ff640 10 monclient: init 2024-03-25T18:10:10.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:10.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:10.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-a con 0x7f13300a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-b con 0x7f133013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-c con 0x7f13301430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330142520 con 0x7f13300a98f0 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300b28a0 con 0x7f13301430c0 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300b2a30 con 0x7f133013f710 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:10.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.185+0000 7f13372ff640 10 monclient(hunting): authenticate will time out at 7254.483376s 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.189+0000 7f133489a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f133013f710 0x7f13301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44340/0 (socket says 172.21.15.84:44340) 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.189+0000 7f133489a640 1 -- 172.21.15.84:0/1816199617 learned_addr learned my addr 172.21.15.84:0/1816199617 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:10.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538284918 0 0) 0x7f13300b2a30 con 0x7f133013f710 2024-03-25T18:10:10.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:10.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:10.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:10.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:10:10.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.198+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c003520 con 0x7f133013f710 2024-03-25T18:10:10.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354453998 0 0) 0x7f131c003520 con 0x7f133013f710 2024-03-25T18:10:10.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 >> v1:172.21.15.157:6789/0 conn(0x7f13301430c0 legacy=0x7f133014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 >> v1:172.21.15.84:6789/0 conn(0x7f13300a98f0 legacy=0x7f13300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:10.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 monclient: found mon.noname-b 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300b44c0 con 0x7f133013f710 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1320001690 con 0x7f133013f710 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f1327fff640 10 monclient: dump: 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.214 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f13372ff640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:10:10.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.213+0000 7f13372ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:10.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13200017f0 con 0x7f133013f710 2024-03-25T18:10:10.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:10.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 1 -- 172.21.15.84:0/1816199617 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1320001c00 con 0x7f133013f710 2024-03-25T18:10:10.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f13372ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:10.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f13372ff640 4 set_mon_vals no callback set 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.225+0000 7f1327fff640 10 monclient: dump: 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:10.226 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 10 monclient: shutdown 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 1 -- 172.21.15.84:0/1816199617 >> v1:172.21.15.84:6790/0 conn(0x7f133013f710 legacy=0x7f13301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 1 -- 172.21.15.84:0/1816199617 shutdown_connections 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 1 -- 172.21.15.84:0/1816199617 >> 172.21.15.84:0/1816199617 conn(0x7f13300b7ea0 msgr2=0x7f13300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:10.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 1 -- 172.21.15.84:0/1816199617 shutdown_connections 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.226+0000 7f13372ff640 1 -- 172.21.15.84:0/1816199617 wait complete. 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 10 monclient: build_initial_monmap 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 10 monclient: monmap: 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 librados: starting msgr at 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 librados: starting objecter 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 Processor -- start 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 -- start start 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 librados: setting wanted keys 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 1 librados: calling monclient init 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 10 monclient: init 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.227+0000 7f13372ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-c con 0x7f13300a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-b con 0x7f133013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 10 monclient(hunting): picked mon.noname-a con 0x7f13301430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330138c10 con 0x7f13301430c0 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330138e60 con 0x7f133013f710 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330139140 con 0x7f13300a98f0 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:10.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.228+0000 7f13372ff640 10 monclient(hunting): authenticate will time out at 7254.525964s 2024-03-25T18:10:10.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.233+0000 7f133589c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f13301430c0 0x7f13301e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:41188/0 (socket says 172.21.15.84:41188) 2024-03-25T18:10:10.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.233+0000 7f133589c640 1 -- 172.21.15.84:0/515716094 learned_addr learned my addr 172.21.15.84:0/515716094 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:10.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.238+0000 7f1325ffb640 10 client.?.objecter ms_handle_connect 0x7f13301430c0 2024-03-25T18:10:10.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.238+0000 7f1325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246810226 0 0) 0x7f1330138c10 con 0x7f13301430c0 2024-03-25T18:10:10.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:10.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:10.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:10.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:10:10.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.245+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1314004eb0 con 0x7f13301430c0 2024-03-25T18:10:10.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.249+0000 7f1325ffb640 10 client.?.objecter ms_handle_connect 0x7f133013f710 2024-03-25T18:10:10.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.249+0000 7f1325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.253+0000 7f1325ffb640 10 client.?.objecter ms_handle_connect 0x7f13300a98f0 2024-03-25T18:10:10.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.253+0000 7f1325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4211180431 0 0) 0x7f1314004eb0 con 0x7f13301430c0 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1330138c10 con 0x7f13301430c0 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1328001c90 con 0x7f13301430c0 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:10.258 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:10.259 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.257+0000 7f1325ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066752833 0 0) 0x7f1330139140 con 0x7f13300a98f0 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:10:10.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.263+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1314004eb0 con 0x7f13300a98f0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1529615453 0 0) 0x7f1330138c10 con 0x7f13301430c0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 >> v1:172.21.15.84:6790/0 conn(0x7f13300a98f0 legacy=0x7f1330137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 >> v1:172.21.15.157:6789/0 conn(0x7f133013f710 legacy=0x7f1330132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 1 monclient: found mon.a 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330139920 con 0x7f13301430c0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:10.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f1325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 10 monclient: _renew_subs 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13301382e0 con 0x7f13301430c0 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 10 monclient: _renew_subs 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300ab8b0 con 0x7f13301430c0 2024-03-25T18:10:10.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.266+0000 7f13372ff640 1 librados: init done 2024-03-25T18:10:10.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.267+0000 7f13372ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.267+0000 7f13372ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.267+0000 7f13372ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:10.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.267+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133013d010 con 0x7f13301430c0 2024-03-25T18:10:10.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.270+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1328003990 con 0x7f13301430c0 2024-03-25T18:10:10.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.270+0000 7f1325ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:10.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.270+0000 7f13277fe640 4 set_mon_vals no callback set 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.567+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1328003da0 con 0x7f13301430c0 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.567+0000 7f1325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.567+0000 7f1325ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.567+0000 7f1325ffb640 10 monclient: dump: 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:10.568 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.568+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1328003f80 con 0x7f13301430c0 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.569+0000 7f133509b640 10 monclient: get_auth_request con 0x7f131406dd40 auth_method 0 2024-03-25T18:10:10.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.569+0000 7f1325ffb640 10 client.4449.objecter ms_handle_connect 0x7f131406dd40 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1328054b60 con 0x7f13301430c0 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 10 client.4449.objecter ms_dispatch 0x7f1330000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T18:10:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1328050670 con 0x7f13301430c0 2024-03-25T18:10:10.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.579+0000 7f1325ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.802+0000 7f13372ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.802+0000 7f13372ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.802+0000 7f13372ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.802+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1330139660 con 0x7f13301430c0 2024-03-25T18:10:10.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.868+0000 7f1325ffb640 1 -- 172.21.15.84:0/515716094 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1330139660 con 0x7f13301430c0 2024-03-25T18:10:10.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.868+0000 7f1325ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.868+0000 7f1325ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:10.869 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:10.869 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 10 librados: watch_flush enter 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 10 librados: watch_flush exit 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 10 client.4449.objecter successfully canceled tick 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 >> v1:172.21.15.157:6800/1801259026 conn(0x7f131406dd40 legacy=0x7f1314070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 10 monclient: shutdown 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:10.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 >> v1:172.21.15.84:6789/0 conn(0x7f13301430c0 legacy=0x7f13301e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 shutdown_connections 2024-03-25T18:10:10.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.870+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 >> 172.21.15.84:0/515716094 conn(0x7f13300b7ea0 msgr2=0x7f1330074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:10.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.871+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 shutdown_connections 2024-03-25T18:10:10.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.871+0000 7f13372ff640 1 -- 172.21.15.84:0/515716094 wait complete. 2024-03-25T18:10:10.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:10.871+0000 7f13372ff640 1 librados: shutdown 2024-03-25T18:10:10.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:17.692 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.805+0000 7fb464993640 10 monclient: get_monmap_and_config 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.805+0000 7fb464993640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.805+0000 7fb464993640 10 monclient: monmap: 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:17.807197+0000 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:17.807197+0000 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:17.807 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.806+0000 7fb464993640 1 Processor -- start 2024-03-25T18:10:17.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.806+0000 7fb464993640 1 -- start start 2024-03-25T18:10:17.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.806+0000 7fb464993640 10 monclient: init 2024-03-25T18:10:17.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.807+0000 7fb464993640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.807+0000 7fb464993640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.807+0000 7fb464993640 10 monclient(hunting): picked mon.noname-c con 0x7fb4600b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.807+0000 7fb464993640 10 monclient(hunting): picked mon.noname-a con 0x7fb4600b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 10 monclient(hunting): picked mon.noname-b con 0x7fb46014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4600af940 con 0x7fb4600b1730 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4600b0a70 con 0x7fb4600b4d60 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4600b0d50 con 0x7fb46014be90 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.808+0000 7fb464993640 10 monclient(hunting): authenticate will time out at 7262.107087s 2024-03-25T18:10:17.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.814+0000 7fb45ed9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb46014be90 0x7fb46014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49274/0 (socket says 172.21.15.84:49274) 2024-03-25T18:10:17.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.814+0000 7fb45ed9d640 1 -- 172.21.15.84:0/2263706805 learned_addr learned my addr 172.21.15.84:0/2263706805 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107802490 0 0) 0x7fb4600af940 con 0x7fb4600b1730 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:10:17.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.827+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb448003520 con 0x7fb4600b1730 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.835+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679092449 0 0) 0x7fb448003520 con 0x7fb4600b1730 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 >> v1:172.21.15.84:6790/0 conn(0x7fb46014be90 legacy=0x7fb46014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 >> v1:172.21.15.157:6789/0 conn(0x7fb4600b4d60 legacy=0x7fb46013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 1 monclient: found mon.noname-a 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:10:17.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4600b1030 con 0x7fb4600b1730 2024-03-25T18:10:17.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb45d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb464993640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:10:17.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.836+0000 7fb464993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:17.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.838+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb454001690 con 0x7fb4600b1730 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.838+0000 7fb45d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.838+0000 7fb45d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.838+0000 7fb45d59a640 10 monclient: dump: 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:17.840 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.838+0000 7fb464993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:17.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb454001920 con 0x7fb4600b1730 2024-03-25T18:10:17.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:17.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 1 -- 172.21.15.84:0/2263706805 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb454001c30 con 0x7fb4600b1730 2024-03-25T18:10:17.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb464993640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb464993640 4 set_mon_vals no callback set 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.850+0000 7fb45d59a640 10 monclient: dump: 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:17.852 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 10 monclient: shutdown 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 1 -- 172.21.15.84:0/2263706805 >> v1:172.21.15.84:6789/0 conn(0x7fb4600b1730 legacy=0x7fb4600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 1 -- 172.21.15.84:0/2263706805 shutdown_connections 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 1 -- 172.21.15.84:0/2263706805 >> 172.21.15.84:0/2263706805 conn(0x7fb4600617a0 msgr2=0x7fb460061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:17.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 1 -- 172.21.15.84:0/2263706805 shutdown_connections 2024-03-25T18:10:17.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.851+0000 7fb464993640 1 -- 172.21.15.84:0/2263706805 wait complete. 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.852+0000 7fb464993640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.852+0000 7fb464993640 10 monclient: monmap: 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:17.855 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 librados: starting msgr at 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 librados: starting objecter 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 Processor -- start 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 -- start start 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 librados: setting wanted keys 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 1 librados: calling monclient init 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 10 monclient: init 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:17.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.853+0000 7fb464993640 10 monclient(hunting): picked mon.noname-a con 0x7fb4600b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 10 monclient(hunting): picked mon.noname-c con 0x7fb4600b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 10 monclient(hunting): picked mon.noname-b con 0x7fb46014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb460140e50 con 0x7fb4600b1730 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb460140930 con 0x7fb46014be90 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb460138760 con 0x7fb4600b4d60 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.854+0000 7fb464993640 10 monclient(hunting): authenticate will time out at 7262.153014s 2024-03-25T18:10:17.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.866+0000 7fb45ed9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb46014be90 0x7fb4601e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:47492/0 (socket says 172.21.15.84:47492) 2024-03-25T18:10:17.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.866+0000 7fb45ed9d640 1 -- 172.21.15.84:0/2604202620 learned_addr learned my addr 172.21.15.84:0/2604202620 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:17.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.866+0000 7fb44effd640 10 client.?.objecter ms_handle_connect 0x7fb4600b1730 2024-03-25T18:10:17.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.866+0000 7fb44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.867+0000 7fb44effd640 10 client.?.objecter ms_handle_connect 0x7fb46014be90 2024-03-25T18:10:17.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.867+0000 7fb44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851963156 0 0) 0x7fb460140e50 con 0x7fb4600b1730 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:10:17.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.869+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb440004eb0 con 0x7fb4600b1730 2024-03-25T18:10:17.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.878+0000 7fb44effd640 10 client.?.objecter ms_handle_connect 0x7fb4600b4d60 2024-03-25T18:10:17.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.878+0000 7fb44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860557570 0 0) 0x7fb440004eb0 con 0x7fb4600b1730 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb460140e50 con 0x7fb4600b1730 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb4500018c0 con 0x7fb4600b1730 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 10 monclient(hunting): dump: 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:17.882 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.880+0000 7fb44effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678377281 0 0) 0x7fb460140930 con 0x7fb46014be90 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:10:17.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.883+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb440004eb0 con 0x7fb46014be90 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1617634035 0 0) 0x7fb460140e50 con 0x7fb4600b1730 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 >> v1:172.21.15.84:6790/0 conn(0x7fb4600b4d60 legacy=0x7fb4601dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 >> v1:172.21.15.157:6789/0 conn(0x7fb46014be90 legacy=0x7fb4601e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 1 monclient: found mon.a 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4601388f0 con 0x7fb4600b1730 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb44effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 10 monclient: _renew_subs 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb460137e90 con 0x7fb4600b1730 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 10 monclient: _renew_subs 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb460132800 con 0x7fb4600b1730 2024-03-25T18:10:17.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.886+0000 7fb464993640 1 librados: init done 2024-03-25T18:10:17.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.888+0000 7fb464993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.888+0000 7fb464993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.888+0000 7fb464993640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:17.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.888+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4600627b0 con 0x7fb4600b1730 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4500035f0 con 0x7fb4600b1730 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb4500039c0 con 0x7fb4600b1730 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb44effd640 10 monclient: dump: 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:17.896 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:17.897 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:17.897 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:17.897 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:17.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:17.894+0000 7fb45cd99640 4 set_mon_vals no callback set 2024-03-25T18:10:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.179+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fb450003b20 con 0x7fb4600b1730 2024-03-25T18:10:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.180+0000 7fb45dd9b640 10 monclient: get_auth_request con 0x7fb44006dd40 auth_method 0 2024-03-25T18:10:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.180+0000 7fb44effd640 10 client.4455.objecter ms_handle_connect 0x7fb44006dd40 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fb450053560 con 0x7fb4600b1730 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 10 client.4455.objecter ms_dispatch 0x7fb460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb45004f070 con 0x7fb4600b1730 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:18.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.188+0000 7fb44effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:18.335 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:15.154 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.345 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:15.164 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.382 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:15.201 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.383 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:15.203 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.389 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:15.173 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.390 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:15.210 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.398 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:15.181 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.413+0000 7fb464993640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.413+0000 7fb464993640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.413+0000 7fb464993640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:18.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.413+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4601401c0 con 0x7fb4600b1730 2024-03-25T18:10:18.417 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:15.236 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.419 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:15.203 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.629+0000 7fb44effd640 1 -- 172.21.15.84:0/2604202620 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fb450050fa0 con 0x7fb4600b1730 2024-03-25T18:10:18.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.629+0000 7fb44effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.629+0000 7fb44effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:18.631 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:18.631 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:18.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 10 librados: watch_flush enter 2024-03-25T18:10:18.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 10 librados: watch_flush exit 2024-03-25T18:10:18.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T18:10:18.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 10 client.4455.objecter successfully canceled tick 2024-03-25T18:10:18.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 >> v1:172.21.15.157:6800/1801259026 conn(0x7fb44006dd40 legacy=0x7fb440070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 10 monclient: shutdown 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 >> v1:172.21.15.84:6789/0 conn(0x7fb4600b1730 legacy=0x7fb460131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 shutdown_connections 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 >> 172.21.15.84:0/2604202620 conn(0x7fb4600617a0 msgr2=0x7fb46013c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 shutdown_connections 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 -- 172.21.15.84:0/2604202620 wait complete. 2024-03-25T18:10:18.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:18.631+0000 7fb464993640 1 librados: shutdown 2024-03-25T18:10:18.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:25.449 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:25.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.566+0000 7f2d941a3640 10 monclient: get_monmap_and_config 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.566+0000 7f2d941a3640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.566+0000 7f2d941a3640 10 monclient: monmap: 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:25.568074+0000 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:25.568074+0000 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:25.568 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.567+0000 7f2d941a3640 1 Processor -- start 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.567+0000 7f2d941a3640 1 -- start start 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.567+0000 7f2d941a3640 10 monclient: init 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-b con 0x7f2d8c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c142520 con 0x7f2d8c1430c0 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0b28a0 con 0x7f2d8c13f710 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0b2a30 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.568+0000 7f2d941a3640 10 monclient(hunting): authenticate will time out at 7269.867408s 2024-03-25T18:10:25.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.576+0000 7f2d9173e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2d8c13f710 0x7f2d8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59854/0 (socket says 172.21.15.84:59854) 2024-03-25T18:10:25.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.576+0000 7f2d9173e640 1 -- 172.21.15.84:0/982959508 learned_addr learned my addr 172.21.15.84:0/982959508 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895412311 0 0) 0x7f2d8c0b28a0 con 0x7f2d8c13f710 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:10:25.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.591+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d78003520 con 0x7f2d8c13f710 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062514150 0 0) 0x7f2d8c0b2a30 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:10:25.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.600+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8c0b28a0 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420623813 0 0) 0x7f2d78003520 con 0x7f2d8c13f710 2024-03-25T18:10:25.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 >> v1:172.21.15.84:6790/0 conn(0x7f2d8c0a98f0 legacy=0x7f2d8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 >> v1:172.21.15.84:6789/0 conn(0x7f2d8c1430c0 legacy=0x7f2d8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 monclient: found mon.noname-c 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0b44c0 con 0x7f2d8c13f710 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d7c001690 con 0x7f2d8c13f710 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d941a3640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d941a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d90f3d640 10 monclient: dump: 2024-03-25T18:10:25.608 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.606+0000 7f2d941a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:25.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d7c0017f0 con 0x7f2d8c13f710 2024-03-25T18:10:25.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 1 -- 172.21.15.84:0/982959508 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d7c001c00 con 0x7f2d8c13f710 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d941a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d941a3640 4 set_mon_vals no callback set 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.618+0000 7f2d90f3d640 10 monclient: dump: 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:25.620 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 10 monclient: shutdown 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 1 -- 172.21.15.84:0/982959508 >> v1:172.21.15.157:6789/0 conn(0x7f2d8c13f710 legacy=0x7f2d8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 1 -- 172.21.15.84:0/982959508 shutdown_connections 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 1 -- 172.21.15.84:0/982959508 >> 172.21.15.84:0/982959508 conn(0x7f2d8c0b7ea0 msgr2=0x7f2d8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 1 -- 172.21.15.84:0/982959508 shutdown_connections 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.619+0000 7f2d941a3640 1 -- 172.21.15.84:0/982959508 wait complete. 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.620+0000 7f2d941a3640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.620+0000 7f2d941a3640 10 monclient: monmap: 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:25.622 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.620+0000 7f2d941a3640 1 librados: starting msgr at 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.620+0000 7f2d941a3640 1 librados: starting objecter 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.620+0000 7f2d941a3640 1 Processor -- start 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 -- start start 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 librados: setting wanted keys 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 librados: calling monclient init 2024-03-25T18:10:25.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient: init 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-b con 0x7f2d8c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c139a40 con 0x7f2d8c1430c0 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c133f20 con 0x7f2d8c13f710 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0ab950 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.621+0000 7f2d941a3640 10 monclient(hunting): authenticate will time out at 7269.920564s 2024-03-25T18:10:25.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.628+0000 7f2d9173e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2d8c13f710 0x7f2d8c133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59906/0 (socket says 172.21.15.84:59906) 2024-03-25T18:10:25.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.628+0000 7f2d9173e640 1 -- 172.21.15.84:0/826434518 learned_addr learned my addr 172.21.15.84:0/826434518 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:25.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.629+0000 7f2d827fc640 10 client.?.objecter ms_handle_connect 0x7f2d8c13f710 2024-03-25T18:10:25.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.629+0000 7f2d827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.637+0000 7f2d827fc640 10 client.?.objecter ms_handle_connect 0x7f2d8c0a98f0 2024-03-25T18:10:25.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.637+0000 7f2d827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052728761 0 0) 0x7f2d8c133f20 con 0x7f2d8c13f710 2024-03-25T18:10:25.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:10:25.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.643+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d70004eb0 con 0x7f2d8c13f710 2024-03-25T18:10:25.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.646+0000 7f2d827fc640 10 client.?.objecter ms_handle_connect 0x7f2d8c1430c0 2024-03-25T18:10:25.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.646+0000 7f2d827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001460897 0 0) 0x7f2d8c0ab950 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:10:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.652+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8c133f20 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162239317 0 0) 0x7f2d8c139a40 con 0x7f2d8c1430c0 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.661+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8c0ab950 con 0x7f2d8c1430c0 2024-03-25T18:10:25.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3111468211 0 0) 0x7f2d70004eb0 con 0x7f2d8c13f710 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d8c139a40 con 0x7f2d8c13f710 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d7c001bc0 con 0x7f2d8c13f710 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 10 monclient(hunting): dump: 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:25.664 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:25.665 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:25.665 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.662+0000 7f2d827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:25.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2856303024 0 0) 0x7f2d8c133f20 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d70004eb0 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d88001930 con 0x7f2d8c0a98f0 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.672+0000 7f2d827fc640 10 monclient(hunting): dump: 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:25.674 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921284641 0 0) 0x7f2d8c139a40 con 0x7f2d8c13f710 2024-03-25T18:10:25.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 >> v1:172.21.15.84:6790/0 conn(0x7f2d8c0a98f0 legacy=0x7f2d8c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 >> v1:172.21.15.84:6789/0 conn(0x7f2d8c1430c0 legacy=0x7f2d8c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 1 monclient: found mon.b 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0abc30 con 0x7f2d8c13f710 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 10 monclient: _renew_subs 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0abf10 con 0x7f2d8c13f710 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 10 monclient: _renew_subs 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c139370 con 0x7f2d8c13f710 2024-03-25T18:10:25.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.674+0000 7f2d941a3640 1 librados: init done 2024-03-25T18:10:25.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.675+0000 7f2d941a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.675+0000 7f2d941a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.675+0000 7f2d941a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:10:25.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.675+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c0627b0 con 0x7f2d8c13f710 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d7c0011f0 con 0x7f2d8c13f710 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2d7c004cb0 con 0x7f2d8c13f710 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 10 monclient: dump: 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:25.956 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d83fff640 4 set_mon_vals no callback set 2024-03-25T18:10:25.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.955+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2d7c01ad00 con 0x7f2d8c13f710 2024-03-25T18:10:25.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.956+0000 7f2d91f3f640 10 monclient: get_auth_request con 0x7f2d7006ded0 auth_method 0 2024-03-25T18:10:25.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.956+0000 7f2d827fc640 10 client.4465.objecter ms_handle_connect 0x7f2d7006ded0 2024-03-25T18:10:25.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f2d7c054970 con 0x7f2d8c13f710 2024-03-25T18:10:25.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 10 client.4465.objecter ms_dispatch 0x7f2d8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:25.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:25.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:25.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:10:25.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d7c0502b0 con 0x7f2d8c13f710 2024-03-25T18:10:25.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:25.960+0000 7f2d827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:26.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.195+0000 7f2d941a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.195+0000 7f2d941a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.195+0000 7f2d941a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:10:26.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.195+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d8c0b7180 con 0x7f2d8c13f710 2024-03-25T18:10:26.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.278+0000 7f2d827fc640 1 -- 172.21.15.84:0/826434518 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f2d7c0523b0 con 0x7f2d8c13f710 2024-03-25T18:10:26.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.278+0000 7f2d827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.278+0000 7f2d827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:26.280 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:26.280 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 10 librados: watch_flush enter 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 10 librados: watch_flush exit 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 >> v1:172.21.15.157:6800/1801259026 conn(0x7f2d7006ded0 legacy=0x7f2d70070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:26.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 10 monclient: shutdown 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 >> v1:172.21.15.157:6789/0 conn(0x7f2d8c13f710 legacy=0x7f2d8c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 shutdown_connections 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 >> 172.21.15.84:0/826434518 conn(0x7f2d8c0b7ea0 msgr2=0x7f2d8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 shutdown_connections 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.280+0000 7f2d941a3640 1 -- 172.21.15.84:0/826434518 wait complete. 2024-03-25T18:10:26.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:26.281+0000 7f2d941a3640 1 librados: shutdown 2024-03-25T18:10:26.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:33.100 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:33.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 10 monclient: get_monmap_and_config 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 10 monclient: build_initial_monmap 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 10 monclient: monmap: 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:33.218513+0000 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:33.218513+0000 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:33.219 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 1 Processor -- start 2024-03-25T18:10:33.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 1 -- start start 2024-03-25T18:10:33.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.217+0000 7f1a8500e640 10 monclient: init 2024-03-25T18:10:33.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:33.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a800a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a8013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a801430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a80142520 con 0x7f1a800a98f0 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a800b28a0 con 0x7f1a801430c0 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a800b2a30 con 0x7f1a8013f710 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:33.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.218+0000 7f1a8500e640 10 monclient(hunting): authenticate will time out at 7277.517670s 2024-03-25T18:10:33.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.224+0000 7f1a7f59e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1a801430c0 0x7f1a8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35824/0 (socket says 172.21.15.84:35824) 2024-03-25T18:10:33.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.224+0000 7f1a7f59e640 1 -- 172.21.15.84:0/3396687376 learned_addr learned my addr 172.21.15.84:0/3396687376 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.232+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313855061 0 0) 0x7f1a800b28a0 con 0x7f1a801430c0 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.232+0000 7f1a7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.233+0000 7f1a7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.233+0000 7f1a7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.233+0000 7f1a7dd9b640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:10:33.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.233+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68003520 con 0x7f1a801430c0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270578296 0 0) 0x7f1a68003520 con 0x7f1a801430c0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 >> v1:172.21.15.84:6790/0 conn(0x7f1a8013f710 legacy=0x7f1a801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 >> v1:172.21.15.84:6789/0 conn(0x7f1a800a98f0 legacy=0x7f1a800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a800b44c0 con 0x7f1a801430c0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:33.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a78001a10 con 0x7f1a801430c0 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a7dd9b640 10 monclient: dump: 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a8500e640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:10:33.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.242+0000 7f1a8500e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:33.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a78001b70 con 0x7f1a801430c0 2024-03-25T18:10:33.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 1 -- 172.21.15.84:0/3396687376 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a78001f80 con 0x7f1a801430c0 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a7dd9b640 10 monclient: dump: 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a8500e640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:33.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.248+0000 7f1a8500e640 4 set_mon_vals no callback set 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 10 monclient: shutdown 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 1 -- 172.21.15.84:0/3396687376 >> v1:172.21.15.157:6789/0 conn(0x7f1a801430c0 legacy=0x7f1a8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 1 -- 172.21.15.84:0/3396687376 shutdown_connections 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 1 -- 172.21.15.84:0/3396687376 >> 172.21.15.84:0/3396687376 conn(0x7f1a800b7ea0 msgr2=0x7f1a800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 1 -- 172.21.15.84:0/3396687376 shutdown_connections 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.249+0000 7f1a8500e640 1 -- 172.21.15.84:0/3396687376 wait complete. 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.250+0000 7f1a8500e640 10 monclient: build_initial_monmap 2024-03-25T18:10:33.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.250+0000 7f1a8500e640 10 monclient: monmap: 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.250+0000 7f1a8500e640 1 librados: starting msgr at 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.250+0000 7f1a8500e640 1 librados: starting objecter 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 Processor -- start 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 -- start start 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 librados: setting wanted keys 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 librados: calling monclient init 2024-03-25T18:10:33.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient: init 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a800a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a8013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a801430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a80139820 con 0x7f1a800a98f0 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a80138f20 con 0x7f1a801430c0 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a800ab9e0 con 0x7f1a8013f710 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:33.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.251+0000 7f1a8500e640 10 monclient(hunting): authenticate will time out at 7277.550812s 2024-03-25T18:10:33.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.260+0000 7f1a7e59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1a8013f710 0x7f1a80135310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57564/0 (socket says 172.21.15.84:57564) 2024-03-25T18:10:33.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.260+0000 7f1a7e59c640 1 -- 172.21.15.84:0/4018942660 learned_addr learned my addr 172.21.15.84:0/4018942660 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:33.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.261+0000 7f1a677fe640 10 client.?.objecter ms_handle_connect 0x7f1a8013f710 2024-03-25T18:10:33.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.261+0000 7f1a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.273+0000 7f1a677fe640 10 client.?.objecter ms_handle_connect 0x7f1a801430c0 2024-03-25T18:10:33.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.273+0000 7f1a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.275+0000 7f1a677fe640 10 client.?.objecter ms_handle_connect 0x7f1a800a98f0 2024-03-25T18:10:33.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.275+0000 7f1a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700463983 0 0) 0x7f1a800ab9e0 con 0x7f1a8013f710 2024-03-25T18:10:33.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:10:33.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.276+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c004eb0 con 0x7f1a8013f710 2024-03-25T18:10:33.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342836248 0 0) 0x7f1a6c004eb0 con 0x7f1a8013f710 2024-03-25T18:10:33.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a800ab9e0 con 0x7f1a8013f710 2024-03-25T18:10:33.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a74001a30 con 0x7f1a8013f710 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 10 monclient(hunting): dump: 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.286+0000 7f1a677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501281555 0 0) 0x7f1a80139820 con 0x7f1a800a98f0 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.291+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c004eb0 con 0x7f1a800a98f0 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199741508 0 0) 0x7f1a80138f20 con 0x7f1a801430c0 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:10:33.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.299+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a80139820 con 0x7f1a801430c0 2024-03-25T18:10:33.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2152719208 0 0) 0x7f1a800ab9e0 con 0x7f1a8013f710 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 >> v1:172.21.15.157:6789/0 conn(0x7f1a801430c0 legacy=0x7f1a801e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 >> v1:172.21.15.84:6789/0 conn(0x7f1a800a98f0 legacy=0x7f1a800ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 1 monclient: found mon.c 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a800abcc0 con 0x7f1a8013f710 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 5 monclient: authenticate success, global_id 4472 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 10 monclient: _renew_subs 2024-03-25T18:10:33.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a800abfa0 con 0x7f1a8013f710 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 10 monclient: _renew_subs 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a80138930 con 0x7f1a8013f710 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.301+0000 7f1a8500e640 1 librados: init done 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.302+0000 7f1a8500e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.302+0000 7f1a8500e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.302+0000 7f1a8500e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:33.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.302+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a800627b0 con 0x7f1a8013f710 2024-03-25T18:10:33.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a74001cb0 con 0x7f1a8013f710 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a74003a60 con 0x7f1a8013f710 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a677fe640 10 monclient: dump: 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:33.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.607+0000 7f1a7d59a640 4 set_mon_vals no callback set 2024-03-25T18:10:33.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.608+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1a74003c40 con 0x7f1a8013f710 2024-03-25T18:10:33.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.609+0000 7f1a7ed9d640 10 monclient: get_auth_request con 0x7f1a6c06de60 auth_method 0 2024-03-25T18:10:33.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.609+0000 7f1a677fe640 10 client.4472.objecter ms_handle_connect 0x7f1a6c06de60 2024-03-25T18:10:33.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1a74054870 con 0x7f1a8013f710 2024-03-25T18:10:33.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 10 client.4472.objecter ms_dispatch 0x7f1a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:33.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:33.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:33.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T18:10:33.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a74050380 con 0x7f1a8013f710 2024-03-25T18:10:33.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.614+0000 7f1a677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.840+0000 7f1a8500e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.840+0000 7f1a8500e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.840+0000 7f1a8500e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.840+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a801390d0 con 0x7f1a8013f710 2024-03-25T18:10:33.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.863+0000 7f1a677fe640 1 -- 172.21.15.84:0/4018942660 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1a801390d0 con 0x7f1a8013f710 2024-03-25T18:10:33.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.863+0000 7f1a677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.863+0000 7f1a677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:33.865 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:33.865 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 10 librados: watch_flush enter 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 10 librados: watch_flush exit 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 10 client.4472.objecter successfully canceled tick 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1a6c06de60 legacy=0x7f1a6c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 10 monclient: shutdown 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 >> v1:172.21.15.84:6790/0 conn(0x7f1a8013f710 legacy=0x7f1a80135310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 shutdown_connections 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 >> 172.21.15.84:0/4018942660 conn(0x7f1a800b7ea0 msgr2=0x7f1a80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 shutdown_connections 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 -- 172.21.15.84:0/4018942660 wait complete. 2024-03-25T18:10:33.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:33.866+0000 7f1a8500e640 1 librados: shutdown 2024-03-25T18:10:33.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:38.214 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:35.033 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.225 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:35.045 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.260 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:35.079 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.273 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:35.057 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.277 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:35.096 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.278 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:35.062 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.287 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:35.107 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.311 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:05:35.095 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.321 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:35.140 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:40.686 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.800+0000 7f5fa132b640 10 monclient: get_monmap_and_config 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.800+0000 7f5fa132b640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.800+0000 7f5fa132b640 10 monclient: monmap: 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:40.801670+0000 2024-03-25T18:10:40.801 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:40.801670+0000 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:40.802 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 1 Processor -- start 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 1 -- start start 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient: init 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:40.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c142520 con 0x7f5f9c1430c0 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0b28a0 con 0x7f5f9c13f710 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0b2a30 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.801+0000 7f5fa132b640 10 monclient(hunting): authenticate will time out at 7285.100672s 2024-03-25T18:10:40.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.805+0000 7f5f9affd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5f9c0a98f0 0x7f5f9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50568/0 (socket says 172.21.15.84:50568) 2024-03-25T18:10:40.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.805+0000 7f5f9affd640 1 -- 172.21.15.84:0/4134078238 learned_addr learned my addr 172.21.15.84:0/4134078238 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:40.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846368912 0 0) 0x7f5f9c0b2a30 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:10:40.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.816+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88003520 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2938386770 0 0) 0x7f5f88003520 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 >> v1:172.21.15.157:6789/0 conn(0x7f5f9c13f710 legacy=0x7f5f9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 >> v1:172.21.15.84:6789/0 conn(0x7f5f9c1430c0 legacy=0x7f5f9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0b44c0 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5f8c001690 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5fa132b640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5fa132b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5f9a7fc640 10 monclient: dump: 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:40.829 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:40.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.827+0000 7f5fa132b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f8c0017f0 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 1 -- 172.21.15.84:0/4134078238 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5f8c001c00 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5f9a7fc640 10 monclient: dump: 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:40.838 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5fa132b640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:40.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5fa132b640 4 set_mon_vals no callback set 2024-03-25T18:10:40.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5fa132b640 10 monclient: shutdown 2024-03-25T18:10:40.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5fa132b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.837+0000 7f5fa132b640 1 -- 172.21.15.84:0/4134078238 >> v1:172.21.15.84:6790/0 conn(0x7f5f9c0a98f0 legacy=0x7f5f9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.838+0000 7f5fa132b640 1 -- 172.21.15.84:0/4134078238 shutdown_connections 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.838+0000 7f5fa132b640 1 -- 172.21.15.84:0/4134078238 >> 172.21.15.84:0/4134078238 conn(0x7f5f9c0b7ea0 msgr2=0x7f5f9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.838+0000 7f5fa132b640 1 -- 172.21.15.84:0/4134078238 shutdown_connections 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.838+0000 7f5fa132b640 1 -- 172.21.15.84:0/4134078238 wait complete. 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 10 monclient: monmap: 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:40.841 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 librados: starting msgr at 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 librados: starting objecter 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 Processor -- start 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 -- start start 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 librados: setting wanted keys 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 1 librados: calling monclient init 2024-03-25T18:10:40.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.839+0000 7f5fa132b640 10 monclient: init 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:40.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c135730 con 0x7f5f9c13f710 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c1358c0 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c135a50 con 0x7f5f9c1430c0 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.840+0000 7f5fa132b640 10 monclient(hunting): authenticate will time out at 7285.139328s 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.843+0000 7f5f9b7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5f9c1430c0 0x7f5f9c1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50594/0 (socket says 172.21.15.84:50594) 2024-03-25T18:10:40.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.843+0000 7f5f9b7fe640 1 -- 172.21.15.84:0/1247156311 learned_addr learned my addr 172.21.15.84:0/1247156311 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:40.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.844+0000 7f5f937fe640 10 client.?.objecter ms_handle_connect 0x7f5f9c1430c0 2024-03-25T18:10:40.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.844+0000 7f5f937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.855+0000 7f5f937fe640 10 client.?.objecter ms_handle_connect 0x7f5f9c0a98f0 2024-03-25T18:10:40.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.855+0000 7f5f937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.860+0000 7f5f937fe640 10 client.?.objecter ms_handle_connect 0x7f5f9c13f710 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.860+0000 7f5f937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522907349 0 0) 0x7f5f9c135a50 con 0x7f5f9c1430c0 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:10:40.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.861+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f7c004eb0 con 0x7f5f9c1430c0 2024-03-25T18:10:40.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124637354 0 0) 0x7f5f9c1358c0 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:10:40.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.864+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c135a50 con 0x7f5f9c0a98f0 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033710377 0 0) 0x7f5f7c004eb0 con 0x7f5f9c1430c0 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f9c1358c0 con 0x7f5f9c1430c0 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5f94001c90 con 0x7f5f9c1430c0 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 10 monclient(hunting): dump: 2024-03-25T18:10:40.871 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:40.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.870+0000 7f5f937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1323915347 0 0) 0x7f5f9c1358c0 con 0x7f5f9c1430c0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 >> v1:172.21.15.157:6789/0 conn(0x7f5f9c0a98f0 legacy=0x7f5f9c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 >> v1:172.21.15.84:6789/0 conn(0x7f5f9c13f710 legacy=0x7f5f9c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 1 monclient: found mon.c 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c1d5000 con 0x7f5f9c1430c0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 10 monclient: _renew_subs 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c1d52b0 con 0x7f5f9c1430c0 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 10 monclient: _renew_subs 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c13c900 con 0x7f5f9c1430c0 2024-03-25T18:10:40.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.880+0000 7f5fa132b640 1 librados: init done 2024-03-25T18:10:40.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.881+0000 7f5fa132b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.881+0000 7f5fa132b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.881+0000 7f5fa132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:40.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:40.881+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c13cbb0 con 0x7f5f9c1430c0 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f94003990 con 0x7f5f9c1430c0 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5f94003bc0 con 0x7f5f9c1430c0 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f937fe640 10 monclient: dump: 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:41.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.180+0000 7f5f99ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.181+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f5f94003de0 con 0x7f5f9c1430c0 2024-03-25T18:10:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.182+0000 7f5f9affd640 10 monclient: get_auth_request con 0x7f5f7c06dd10 auth_method 0 2024-03-25T18:10:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.182+0000 7f5f937fe640 10 client.4478.objecter ms_handle_connect 0x7f5f7c06dd10 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f5f94054960 con 0x7f5f9c1430c0 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 10 client.4478.objecter ms_dispatch 0x7f5f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:10:41.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f94050470 con 0x7f5f9c1430c0 2024-03-25T18:10:41.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:41.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.187+0000 7f5f937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:41.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.420+0000 7f5fa132b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.420+0000 7f5fa132b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.420+0000 7f5fa132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:10:41.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.420+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f9c1371c0 con 0x7f5f9c1430c0 2024-03-25T18:10:41.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.461+0000 7f5f937fe640 1 -- 172.21.15.84:0/1247156311 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f5f940523a0 con 0x7f5f9c1430c0 2024-03-25T18:10:41.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.461+0000 7f5f937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.461+0000 7f5f937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:41.463 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:41.463 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 10 librados: watch_flush enter 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 10 librados: watch_flush exit 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 >> v1:172.21.15.157:6800/1801259026 conn(0x7f5f7c06dd10 legacy=0x7f5f7c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 10 monclient: shutdown 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 >> v1:172.21.15.84:6790/0 conn(0x7f5f9c1430c0 legacy=0x7f5f9c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 shutdown_connections 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 >> 172.21.15.84:0/1247156311 conn(0x7f5f9c0b7ea0 msgr2=0x7f5f9c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:41.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 shutdown_connections 2024-03-25T18:10:41.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 -- 172.21.15.84:0/1247156311 wait complete. 2024-03-25T18:10:41.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:41.463+0000 7f5fa132b640 1 librados: shutdown 2024-03-25T18:10:41.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:48.282 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:48.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.395+0000 7fcefe6a8640 10 monclient: get_monmap_and_config 2024-03-25T18:10:48.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.395+0000 7fcefe6a8640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.395+0000 7fcefe6a8640 10 monclient: monmap: 2024-03-25T18:10:48.395 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:48.395761+0000 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:48.395761+0000 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:48.396 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 1 Processor -- start 2024-03-25T18:10:48.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 1 -- start start 2024-03-25T18:10:48.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient: init 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcef80b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcef80b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcef814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80af940 con 0x7fcef80b1730 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80b0a70 con 0x7fcef80b4d60 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80b0d50 con 0x7fcef814be90 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.396+0000 7fcefe6a8640 10 monclient(hunting): authenticate will time out at 7292.694768s 2024-03-25T18:10:48.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.402+0000 7fcefcc45640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcef814be90 0x7fcef814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55360/0 (socket says 172.21.15.84:55360) 2024-03-25T18:10:48.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.403+0000 7fcefcc45640 1 -- 172.21.15.84:0/4198426295 learned_addr learned my addr 172.21.15.84:0/4198426295 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:48.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848633469 0 0) 0x7fcef80b0d50 con 0x7fcef814be90 2024-03-25T18:10:48.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:10:48.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.411+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003520 con 0x7fcef814be90 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2100909320 0 0) 0x7fcee0003520 con 0x7fcef814be90 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 >> v1:172.21.15.157:6789/0 conn(0x7fcef80b4d60 legacy=0x7fcef813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 >> v1:172.21.15.84:6789/0 conn(0x7fcef80b1730 legacy=0x7fcef80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef80b1030 con 0x7fcef814be90 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fcef0001a10 con 0x7fcef814be90 2024-03-25T18:10:48.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcef6ffd640 10 monclient: dump: 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcefe6a8640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:10:48.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.419+0000 7fcefe6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:48.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef0001b70 con 0x7fcef814be90 2024-03-25T18:10:48.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 1 -- 172.21.15.84:0/4198426295 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fcef0001f80 con 0x7fcef814be90 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcef6ffd640 10 monclient: dump: 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:48.427 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:48.428 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcefe6a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:48.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.426+0000 7fcefe6a8640 4 set_mon_vals no callback set 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 10 monclient: shutdown 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 1 -- 172.21.15.84:0/4198426295 >> v1:172.21.15.84:6790/0 conn(0x7fcef814be90 legacy=0x7fcef814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 1 -- 172.21.15.84:0/4198426295 shutdown_connections 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 1 -- 172.21.15.84:0/4198426295 >> 172.21.15.84:0/4198426295 conn(0x7fcef80617a0 msgr2=0x7fcef8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 1 -- 172.21.15.84:0/4198426295 shutdown_connections 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.427+0000 7fcefe6a8640 1 -- 172.21.15.84:0/4198426295 wait complete. 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.428+0000 7fcefe6a8640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.428+0000 7fcefe6a8640 10 monclient: monmap: 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:48.430 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.428+0000 7fcefe6a8640 1 librados: starting msgr at 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.428+0000 7fcefe6a8640 1 librados: starting objecter 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.428+0000 7fcefe6a8640 1 Processor -- start 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 -- start start 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 librados: setting wanted keys 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 librados: calling monclient init 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient: init 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:48.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcef80b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcef80b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcef814be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef8137b10 con 0x7fcef814be90 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef81377e0 con 0x7fcef80b1730 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef8137fa0 con 0x7fcef80b4d60 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.429+0000 7fcefe6a8640 10 monclient(hunting): authenticate will time out at 7292.727769s 2024-03-25T18:10:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.442+0000 7fcef77fe640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7fcef80b4d60 legacy=0x7fcef81daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:10:48.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.442+0000 7fcef77fe640 1 -- >> v1:172.21.15.84:6790/0 conn(0x7fcef80b4d60 legacy=0x7fcef81daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:10:48.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.442+0000 7fcef77fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcef80b4d60 0x7fcef81daed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T18:10:48.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.442+0000 7fcef77fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcef80b4d60 0x7fcef81daed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:10:48.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.442+0000 7fcef4ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6790/0 2024-03-25T18:10:48.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.449+0000 7fcefcc45640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fcef814be90 0x7fcef81e3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56904/0 (socket says 172.21.15.84:56904) 2024-03-25T18:10:48.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.449+0000 7fcefcc45640 1 -- 172.21.15.84:0/169132925 learned_addr learned my addr 172.21.15.84:0/169132925 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:48.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.450+0000 7fcef4ff9640 10 client.?.objecter ms_handle_connect 0x7fcef814be90 2024-03-25T18:10:48.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.450+0000 7fcef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.456+0000 7fcef4ff9640 10 client.?.objecter ms_handle_connect 0x7fcef80b1730 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.456+0000 7fcef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186918059 0 0) 0x7fcef8137b10 con 0x7fcef814be90 2024-03-25T18:10:48.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:10:48.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.462+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4004eb0 con 0x7fcef814be90 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163118697 0 0) 0x7fcef81377e0 con 0x7fcef80b1730 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:10:48.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.467+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef8137b10 con 0x7fcef80b1730 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235796793 0 0) 0x7fcee4004eb0 con 0x7fcef814be90 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcef81377e0 con 0x7fcef814be90 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fcef0001ec0 con 0x7fcef814be90 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:48.477 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:48.477 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.475+0000 7fcef4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244978939 0 0) 0x7fcef8137b10 con 0x7fcef80b1730 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee4004eb0 con 0x7fcef80b1730 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fcee80018b0 con 0x7fcef80b1730 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.486+0000 7fcef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.487 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.488 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:48.488 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:48.488 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:48.488 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1637829809 0 0) 0x7fcef81377e0 con 0x7fcef814be90 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 >> v1:172.21.15.84:6790/0 conn(0x7fcef80b4d60 legacy=0x7fcef81daed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 >> v1:172.21.15.157:6789/0 conn(0x7fcef80b1730 legacy=0x7fcef8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 1 monclient: found mon.a 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8142030 con 0x7fcef814be90 2024-03-25T18:10:48.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 10 monclient: _renew_subs 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef8140840 con 0x7fcef814be90 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 10 monclient: _renew_subs 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef81357a0 con 0x7fcef814be90 2024-03-25T18:10:48.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.501+0000 7fcefe6a8640 1 librados: init done 2024-03-25T18:10:48.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.502+0000 7fcefe6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.502+0000 7fcefe6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.502+0000 7fcefe6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:48.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.502+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef80627b0 con 0x7fcef814be90 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef00020e0 con 0x7fcef814be90 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fcef0004fb0 con 0x7fcef814be90 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef4ff9640 10 monclient: dump: 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:48.838 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.837+0000 7fcef67fc640 4 set_mon_vals no callback set 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.838+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fcef001af80 con 0x7fcef814be90 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.839+0000 7fcef7fff640 10 monclient: get_auth_request con 0x7fcee406de90 auth_method 0 2024-03-25T18:10:48.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.839+0000 7fcef4ff9640 10 client.4479.objecter ms_handle_connect 0x7fcee406de90 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fcef0054ad0 con 0x7fcef814be90 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 10 client.4479.objecter ms_dispatch 0x7fcef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcef0050410 con 0x7fcef814be90 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:48.844+0000 7fcef4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:49.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.075+0000 7fcefe6a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.075+0000 7fcefe6a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.075+0000 7fcefe6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:49.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.075+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcef80b75b0 con 0x7fcef814be90 2024-03-25T18:10:49.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.134+0000 7fcef4ff9640 1 -- 172.21.15.84:0/169132925 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fcef0052510 con 0x7fcef814be90 2024-03-25T18:10:49.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.134+0000 7fcef4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.134+0000 7fcef4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:49.134 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:49.134 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.136+0000 7fcefe6a8640 10 librados: watch_flush enter 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 10 librados: watch_flush exit 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 10 client.4479.objecter successfully canceled tick 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 >> v1:172.21.15.157:6800/1801259026 conn(0x7fcee406de90 legacy=0x7fcee4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 10 monclient: shutdown 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 >> v1:172.21.15.84:6789/0 conn(0x7fcef814be90 legacy=0x7fcef81e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 shutdown_connections 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 >> 172.21.15.84:0/169132925 conn(0x7fcef80617a0 msgr2=0x7fcef813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 shutdown_connections 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 -- 172.21.15.84:0/169132925 wait complete. 2024-03-25T18:10:49.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:49.137+0000 7fcefe6a8640 1 librados: shutdown 2024-03-25T18:10:49.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:55.956 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.069+0000 7f07a8fb5640 10 monclient: get_monmap_and_config 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.069+0000 7f07a8fb5640 10 monclient: build_initial_monmap 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.069+0000 7f07a8fb5640 10 monclient: monmap: 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:10:56.069804+0000 2024-03-25T18:10:56.070 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:10:56.069804+0000 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 1 Processor -- start 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 1 -- start start 2024-03-25T18:10:56.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient: init 2024-03-25T18:10:56.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:56.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:56.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f07a40a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f07a413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f07a41430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4142520 con 0x7f07a40a98f0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b28a0 con 0x7f07a413f710 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b2a30 con 0x7f07a41430c0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.070+0000 7f07a8fb5640 10 monclient(hunting): authenticate will time out at 7300.368786s 2024-03-25T18:10:56.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.073+0000 7f07a259c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f07a40a98f0 0x7f07a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44034/0 (socket says 172.21.15.84:44034) 2024-03-25T18:10:56.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.073+0000 7f07a259c640 1 -- 172.21.15.84:0/1189054444 learned_addr learned my addr 172.21.15.84:0/1189054444 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.081+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638720023 0 0) 0x7f07a4142520 con 0x7f07a40a98f0 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.081+0000 7f07a1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.081+0000 7f07a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.081+0000 7f07a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.081+0000 7f07a1d9b640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.082+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790003520 con 0x7f07a40a98f0 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976564926 0 0) 0x7f0790003520 con 0x7f07a40a98f0 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 >> v1:172.21.15.84:6790/0 conn(0x7f07a41430c0 legacy=0x7f07a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 >> v1:172.21.15.157:6789/0 conn(0x7f07a413f710 legacy=0x7f07a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b44c0 con 0x7f07a40a98f0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a8fb5640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a8fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0794001690 con 0x7f07a40a98f0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a1d9b640 10 monclient: dump: 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.088+0000 7f07a8fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0794001960 con 0x7f07a40a98f0 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 1 -- 172.21.15.84:0/1189054444 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0794001cf0 con 0x7f07a40a98f0 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a1d9b640 10 monclient: dump: 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.100 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a8fb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.098+0000 7f07a8fb5640 4 set_mon_vals no callback set 2024-03-25T18:10:56.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 10 monclient: shutdown 2024-03-25T18:10:56.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:56.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1189054444 >> v1:172.21.15.84:6789/0 conn(0x7f07a40a98f0 legacy=0x7f07a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1189054444 shutdown_connections 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1189054444 >> 172.21.15.84:0/1189054444 conn(0x7f07a40b7ea0 msgr2=0x7f07a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1189054444 shutdown_connections 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.099+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1189054444 wait complete. 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.100+0000 7f07a8fb5640 10 monclient: build_initial_monmap 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.100+0000 7f07a8fb5640 10 monclient: monmap: 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:10:56.102 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.100+0000 7f07a8fb5640 1 librados: starting msgr at 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.100+0000 7f07a8fb5640 1 librados: starting objecter 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.100+0000 7f07a8fb5640 1 Processor -- start 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 1 -- start start 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 1 librados: setting wanted keys 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 1 librados: calling monclient init 2024-03-25T18:10:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 10 monclient: init 2024-03-25T18:10:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f07a40a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.101+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f07a413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f07a41430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b6440 con 0x7f07a41430c0 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b6720 con 0x7f07a413f710 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b78a0 con 0x7f07a40a98f0 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.102+0000 7f07a8fb5640 10 monclient(hunting): authenticate will time out at 7300.400308s 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.119+0000 7f07a2d9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f07a41430c0 0x7f07a41d7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44060/0 (socket says 172.21.15.84:44060) 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.119+0000 7f07a2d9d640 1 -- 172.21.15.84:0/1572868918 learned_addr learned my addr 172.21.15.84:0/1572868918 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:10:56.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.121+0000 7f079b7fe640 10 client.?.objecter ms_handle_connect 0x7f07a41430c0 2024-03-25T18:10:56.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.121+0000 7f079b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.126+0000 7f079b7fe640 10 client.?.objecter ms_handle_connect 0x7f07a413f710 2024-03-25T18:10:56.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.126+0000 7f079b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349812154 0 0) 0x7f07a40b6440 con 0x7f07a41430c0 2024-03-25T18:10:56.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:10:56.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.134+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784004eb0 con 0x7f07a41430c0 2024-03-25T18:10:56.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.139+0000 7f079b7fe640 10 client.?.objecter ms_handle_connect 0x7f07a40a98f0 2024-03-25T18:10:56.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.139+0000 7f079b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753211702 0 0) 0x7f07a40b6720 con 0x7f07a413f710 2024-03-25T18:10:56.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:10:56.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.146+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a40b6440 con 0x7f07a413f710 2024-03-25T18:10:56.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171124728 0 0) 0x7f07a40b78a0 con 0x7f07a40a98f0 2024-03-25T18:10:56.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:10:56.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.150+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a40b6720 con 0x7f07a40a98f0 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711302245 0 0) 0x7f0784004eb0 con 0x7f07a41430c0 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a40b78a0 con 0x7f07a41430c0 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f079c001ad0 con 0x7f07a41430c0 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:56.162 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.161+0000 7f079b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:56.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1282987611 0 0) 0x7f07a40b6720 con 0x7f07a40a98f0 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0784004eb0 con 0x7f07a40a98f0 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0794001650 con 0x7f07a40a98f0 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.169+0000 7f079b7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.170 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.171 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.171 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.171 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.171 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1404139053 0 0) 0x7f07a40b6440 con 0x7f07a413f710 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a40b6720 con 0x7f07a413f710 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f078c001810 con 0x7f07a413f710 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.171+0000 7f079b7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.172 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.173 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.177+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3224678949 0 0) 0x7f07a40b78a0 con 0x7f07a41430c0 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 >> v1:172.21.15.84:6790/0 conn(0x7f07a40a98f0 legacy=0x7f07a40b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 >> v1:172.21.15.157:6789/0 conn(0x7f07a413f710 legacy=0x7f07a41d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 1 monclient: found mon.a 2024-03-25T18:10:56.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a413bb30 con 0x7f07a41430c0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f079b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 10 monclient: _renew_subs 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a41e2fe0 con 0x7f07a41430c0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 10 monclient: _renew_subs 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a41e3350 con 0x7f07a41430c0 2024-03-25T18:10:56.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.178+0000 7f07a8fb5640 1 librados: init done 2024-03-25T18:10:56.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.179+0000 7f07a8fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.179+0000 7f07a8fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.179+0000 7f07a8fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.179+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a4074cc0 con 0x7f07a41430c0 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f079c001fe0 con 0x7f07a41430c0 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f079c004d80 con 0x7f07a41430c0 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f079b7fe640 10 monclient: dump: 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:10:56.479 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:10:56.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.478+0000 7f07a159a640 4 set_mon_vals no callback set 2024-03-25T18:10:56.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.479+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f079c01ad50 con 0x7f07a41430c0 2024-03-25T18:10:56.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.480+0000 7f07a259c640 10 monclient: get_auth_request con 0x7f078406e0b0 auth_method 0 2024-03-25T18:10:56.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.480+0000 7f079b7fe640 10 client.4485.objecter ms_handle_connect 0x7f078406e0b0 2024-03-25T18:10:56.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f079c054a30 con 0x7f07a41430c0 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 10 client.4485.objecter ms_dispatch 0x7f07a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f079c050540 con 0x7f07a41430c0 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.484+0000 7f079b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:56.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.709+0000 7f07a8fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.709+0000 7f07a8fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.709+0000 7f07a8fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:10:56.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.709+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07a41d7590 con 0x7f07a41430c0 2024-03-25T18:10:56.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.752+0000 7f079b7fe640 1 -- 172.21.15.84:0/1572868918 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f079c052470 con 0x7f07a41430c0 2024-03-25T18:10:56.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.752+0000 7f079b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.752+0000 7f079b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:56.753 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:10:56.753 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 10 librados: watch_flush enter 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 10 librados: watch_flush exit 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 10 client.4485.objecter successfully canceled tick 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 >> v1:172.21.15.157:6800/1801259026 conn(0x7f078406e0b0 legacy=0x7f0784070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 10 monclient: shutdown 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 >> v1:172.21.15.84:6789/0 conn(0x7f07a41430c0 legacy=0x7f07a41d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 shutdown_connections 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 >> 172.21.15.84:0/1572868918 conn(0x7f07a40b7ea0 msgr2=0x7f07a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 shutdown_connections 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 -- 172.21.15.84:0/1572868918 wait complete. 2024-03-25T18:10:56.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:10:56.755+0000 7f07a8fb5640 1 librados: shutdown 2024-03-25T18:10:56.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:03.250 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:00.069 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.302 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:00.120 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.329 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:00.148 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.347 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:00.131 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.353 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:00.137 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.365 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:00.184 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.371 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:00.190 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.390 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:00.174 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.411 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:00.230 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.576 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.686+0000 7fda9af1c640 10 monclient: get_monmap_and_config 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.686+0000 7fda9af1c640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.686+0000 7fda9af1c640 10 monclient: monmap: 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:03.687163+0000 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:03.687163+0000 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:03.687 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:03.688 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:03.688 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:03.688 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.687+0000 7fda9af1c640 1 Processor -- start 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.687+0000 7fda9af1c640 1 -- start start 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.687+0000 7fda9af1c640 10 monclient: init 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:03.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-c con 0x7fda940a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-b con 0x7fda9413f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-a con 0x7fda941430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda94142520 con 0x7fda941430c0 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda940b28a0 con 0x7fda940a98f0 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda940b2a30 con 0x7fda9413f710 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.688+0000 7fda9af1c640 10 monclient(hunting): authenticate will time out at 7307.986408s 2024-03-25T18:11:03.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.699+0000 7fda93fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fda9413f710 0x7fda941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49688/0 (socket says 172.21.15.84:49688) 2024-03-25T18:11:03.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.699+0000 7fda93fff640 1 -- 172.21.15.84:0/1294248568 learned_addr learned my addr 172.21.15.84:0/1294248568 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:03.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171677956 0 0) 0x7fda94142520 con 0x7fda941430c0 2024-03-25T18:11:03.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84003520 con 0x7fda941430c0 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745056485 0 0) 0x7fda940b2a30 con 0x7fda9413f710 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:11:03.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.719+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94142520 con 0x7fda9413f710 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239815235 0 0) 0x7fda940b28a0 con 0x7fda940a98f0 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:11:03.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.727+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda940b2a30 con 0x7fda940a98f0 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.735+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988910772 0 0) 0x7fda84003520 con 0x7fda941430c0 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 >> v1:172.21.15.84:6790/0 conn(0x7fda9413f710 legacy=0x7fda941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 >> v1:172.21.15.157:6789/0 conn(0x7fda940a98f0 legacy=0x7fda940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 1 monclient: found mon.noname-a 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940b44c0 con 0x7fda941430c0 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fda8c001a10 con 0x7fda941430c0 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda937fe640 10 monclient: dump: 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:03.737 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda9af1c640 5 monclient: authenticate success, global_id 4488 2024-03-25T18:11:03.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.736+0000 7fda9af1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda8c001b70 con 0x7fda941430c0 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 1 -- 172.21.15.84:0/1294248568 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fda8c001f80 con 0x7fda941430c0 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda9af1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda9af1c640 4 set_mon_vals no callback set 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.756+0000 7fda937fe640 10 monclient: dump: 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:03.758 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 10 monclient: shutdown 2024-03-25T18:11:03.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 1 -- 172.21.15.84:0/1294248568 >> v1:172.21.15.84:6789/0 conn(0x7fda941430c0 legacy=0x7fda9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 1 -- 172.21.15.84:0/1294248568 shutdown_connections 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 1 -- 172.21.15.84:0/1294248568 >> 172.21.15.84:0/1294248568 conn(0x7fda940b7ea0 msgr2=0x7fda940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 1 -- 172.21.15.84:0/1294248568 shutdown_connections 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.757+0000 7fda9af1c640 1 -- 172.21.15.84:0/1294248568 wait complete. 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.758+0000 7fda9af1c640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.758+0000 7fda9af1c640 10 monclient: monmap: 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 librados: starting msgr at 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 librados: starting objecter 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 Processor -- start 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 -- start start 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 librados: setting wanted keys 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 1 librados: calling monclient init 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.759+0000 7fda9af1c640 10 monclient: init 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-a con 0x7fda940a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-c con 0x7fda9413f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient(hunting): picked mon.noname-b con 0x7fda941430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda94139830 con 0x7fda940a98f0 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda941db7d0 con 0x7fda941430c0 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda941db410 con 0x7fda9413f710 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.760+0000 7fda9af1c640 10 monclient(hunting): authenticate will time out at 7308.058322s 2024-03-25T18:11:03.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.774+0000 7fda994b9640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fda941430c0 0x7fda941d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35486/0 (socket says 172.21.15.84:35486) 2024-03-25T18:11:03.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.774+0000 7fda994b9640 1 -- 172.21.15.84:0/2798409365 learned_addr learned my addr 172.21.15.84:0/2798409365 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:03.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.776+0000 7fda917fa640 10 client.?.objecter ms_handle_connect 0x7fda940a98f0 2024-03-25T18:11:03.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.776+0000 7fda917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.776+0000 7fda917fa640 10 client.?.objecter ms_handle_connect 0x7fda941430c0 2024-03-25T18:11:03.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.776+0000 7fda917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.785+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824156965 0 0) 0x7fda94139830 con 0x7fda940a98f0 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.785+0000 7fda917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.785+0000 7fda917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.785+0000 7fda917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.785+0000 7fda917fa640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:11:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.786+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda78004eb0 con 0x7fda940a98f0 2024-03-25T18:11:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197378472 0 0) 0x7fda941db7d0 con 0x7fda941430c0 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.787+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94139830 con 0x7fda941430c0 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.788+0000 7fda917fa640 10 client.?.objecter ms_handle_connect 0x7fda9413f710 2024-03-25T18:11:03.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.788+0000 7fda917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589189788 0 0) 0x7fda94139830 con 0x7fda941430c0 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda941db7d0 con 0x7fda941430c0 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fda8c001ec0 con 0x7fda941430c0 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 10 monclient(hunting): dump: 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:03.809 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.808+0000 7fda917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57093670 0 0) 0x7fda78004eb0 con 0x7fda940a98f0 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda94139830 con 0x7fda940a98f0 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fda880018f0 con 0x7fda940a98f0 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): dump: 2024-03-25T18:11:03.811 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:03.812 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662926491 0 0) 0x7fda941db410 con 0x7fda9413f710 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:11:03.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.811+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda78004eb0 con 0x7fda9413f710 2024-03-25T18:11:03.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2753002220 0 0) 0x7fda94139830 con 0x7fda940a98f0 2024-03-25T18:11:03.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 >> v1:172.21.15.84:6790/0 conn(0x7fda9413f710 legacy=0x7fda940b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 >> v1:172.21.15.157:6789/0 conn(0x7fda941430c0 legacy=0x7fda941d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 1 monclient: found mon.a 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda941db5a0 con 0x7fda940a98f0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 10 monclient: _renew_subs 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda940b1480 con 0x7fda940a98f0 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:03.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 10 monclient: _renew_subs 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda940b0b10 con 0x7fda940a98f0 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.818+0000 7fda9af1c640 1 librados: init done 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.819+0000 7fda9af1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.819+0000 7fda9af1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.819+0000 7fda9af1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:03.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:03.819+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda940b1780 con 0x7fda940a98f0 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda88003610 con 0x7fda940a98f0 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fda88003920 con 0x7fda940a98f0 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:04.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda917fa640 10 monclient: dump: 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:04.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.113+0000 7fda92ffd640 4 set_mon_vals no callback set 2024-03-25T18:11:04.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.114+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fda88019930 con 0x7fda940a98f0 2024-03-25T18:11:04.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.115+0000 7fda93fff640 10 monclient: get_auth_request con 0x7fda7806df50 auth_method 0 2024-03-25T18:11:04.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.115+0000 7fda917fa640 10 client.4491.objecter ms_handle_connect 0x7fda7806df50 2024-03-25T18:11:04.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fda88053650 con 0x7fda940a98f0 2024-03-25T18:11:04.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 10 client.4491.objecter ms_dispatch 0x7fda94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:04.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:04.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:04.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:11:04.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda8804ef90 con 0x7fda940a98f0 2024-03-25T18:11:04.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:04.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.120+0000 7fda917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:04.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.351+0000 7fda9af1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.351+0000 7fda9af1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.351+0000 7fda9af1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:04.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.351+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda941db960 con 0x7fda940a98f0 2024-03-25T18:11:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.423+0000 7fda917fa640 1 -- 172.21.15.84:0/2798409365 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fda88051090 con 0x7fda940a98f0 2024-03-25T18:11:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.423+0000 7fda917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.423+0000 7fda917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:04.424 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:04.424 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 10 librados: watch_flush enter 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 10 librados: watch_flush exit 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 >> v1:172.21.15.157:6800/1801259026 conn(0x7fda7806df50 legacy=0x7fda78070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 10 monclient: shutdown 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 >> v1:172.21.15.84:6789/0 conn(0x7fda940a98f0 legacy=0x7fda941dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.426+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 shutdown_connections 2024-03-25T18:11:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.427+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 >> 172.21.15.84:0/2798409365 conn(0x7fda940b7ea0 msgr2=0x7fda94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.427+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 shutdown_connections 2024-03-25T18:11:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.427+0000 7fda9af1c640 1 -- 172.21.15.84:0/2798409365 wait complete. 2024-03-25T18:11:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:04.427+0000 7fda9af1c640 1 librados: shutdown 2024-03-25T18:11:04.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:11.247 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.358+0000 7f01319cf640 10 monclient: get_monmap_and_config 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.358+0000 7f01319cf640 10 monclient: build_initial_monmap 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.358+0000 7f01319cf640 10 monclient: monmap: 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:11.360243+0000 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:11.360243+0000 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.360 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:11.361 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:11.361 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:11.361 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.359+0000 7f01319cf640 1 Processor -- start 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.359+0000 7f01319cf640 1 -- start start 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.359+0000 7f01319cf640 10 monclient: init 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-a con 0x7f012c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-b con 0x7f012c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:11.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-c con 0x7f012c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:11.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c142520 con 0x7f012c0a98f0 2024-03-25T18:11:11.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c0b28a0 con 0x7f012c1430c0 2024-03-25T18:11:11.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c0b2a30 con 0x7f012c13f710 2024-03-25T18:11:11.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:11.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.360+0000 7f01319cf640 10 monclient(hunting): authenticate will time out at 7315.659357s 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.369+0000 7f012b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f012c1430c0 0x7f012c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:38694/0 (socket says 172.21.15.84:38694) 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.369+0000 7f012b7fe640 1 -- 172.21.15.84:0/3873239776 learned_addr learned my addr 172.21.15.84:0/3873239776 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:11.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436668056 0 0) 0x7f012c0b28a0 con 0x7f012c1430c0 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.383+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118003520 con 0x7f012c1430c0 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554750857 0 0) 0x7f012c0b2a30 con 0x7f012c13f710 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:11:11.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.384+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c0b28a0 con 0x7f012c13f710 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.388+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427697166 0 0) 0x7f012c142520 con 0x7f012c0a98f0 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.389+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.389+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.389+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.389+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:11:11.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.389+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c0b2a30 con 0x7f012c0a98f0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478398015 0 0) 0x7f0118003520 con 0x7f012c1430c0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 >> v1:172.21.15.84:6790/0 conn(0x7f012c13f710 legacy=0x7f012c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 >> v1:172.21.15.84:6789/0 conn(0x7f012c0a98f0 legacy=0x7f012c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0b44c0 con 0x7f012c1430c0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0124001a10 con 0x7f012c1430c0 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f012a7fc640 10 monclient: dump: 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f01319cf640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:11:11.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.401+0000 7f01319cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:11.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0124001b70 con 0x7f012c1430c0 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 1 -- 172.21.15.84:0/3873239776 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0124001f80 con 0x7f012c1430c0 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f01319cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f01319cf640 4 set_mon_vals no callback set 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.426+0000 7f012a7fc640 10 monclient: dump: 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.428 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.429 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:11.429 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:11.429 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:11.429 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 10 monclient: shutdown 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 1 -- 172.21.15.84:0/3873239776 >> v1:172.21.15.157:6789/0 conn(0x7f012c1430c0 legacy=0x7f012c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 1 -- 172.21.15.84:0/3873239776 shutdown_connections 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 1 -- 172.21.15.84:0/3873239776 >> 172.21.15.84:0/3873239776 conn(0x7f012c0b7ea0 msgr2=0x7f012c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 1 -- 172.21.15.84:0/3873239776 shutdown_connections 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.427+0000 7f01319cf640 1 -- 172.21.15.84:0/3873239776 wait complete. 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 10 monclient: build_initial_monmap 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 10 monclient: monmap: 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:11.431 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 librados: starting msgr at 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 librados: starting objecter 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 Processor -- start 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 -- start start 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 librados: setting wanted keys 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 1 librados: calling monclient init 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.428+0000 7f01319cf640 10 monclient: init 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:11.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-a con 0x7f012c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-c con 0x7f012c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient(hunting): picked mon.noname-b con 0x7f012c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c13c2b0 con 0x7f012c0a98f0 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c136f10 con 0x7f012c1430c0 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c1370a0 con 0x7f012c13f710 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:11.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.429+0000 7f01319cf640 10 monclient(hunting): authenticate will time out at 7315.728511s 2024-03-25T18:11:11.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.443+0000 7f012b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f012c1430c0 0x7f012c133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:38742/0 (socket says 172.21.15.84:38742) 2024-03-25T18:11:11.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.443+0000 7f012b7fe640 1 -- 172.21.15.84:0/3995773519 learned_addr learned my addr 172.21.15.84:0/3995773519 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:11.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.444+0000 7f0128ff9640 10 client.?.objecter ms_handle_connect 0x7f012c1430c0 2024-03-25T18:11:11.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.444+0000 7f0128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.448+0000 7f0128ff9640 10 client.?.objecter ms_handle_connect 0x7f012c13f710 2024-03-25T18:11:11.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.448+0000 7f0128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.454+0000 7f0128ff9640 10 client.?.objecter ms_handle_connect 0x7f012c0a98f0 2024-03-25T18:11:11.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.454+0000 7f0128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.455+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451139450 0 0) 0x7f012c1370a0 con 0x7f012c13f710 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.456+0000 7f0128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.456+0000 7f0128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.456+0000 7f0128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.456+0000 7f0128ff9640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:11:11.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.456+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c004eb0 con 0x7f012c13f710 2024-03-25T18:11:11.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142476738 0 0) 0x7f012c136f10 con 0x7f012c1430c0 2024-03-25T18:11:11.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:11:11.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.457+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c1370a0 con 0x7f012c1430c0 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.462+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673356292 0 0) 0x7f010c004eb0 con 0x7f012c13f710 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.462+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f012c136f10 con 0x7f012c13f710 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0114001970 con 0x7f012c13f710 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): dump: 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:11.464 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215192222 0 0) 0x7f012c13c2b0 con 0x7f012c0a98f0 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:11:11.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.463+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c004eb0 con 0x7f012c0a98f0 2024-03-25T18:11:11.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3420236720 0 0) 0x7f012c1370a0 con 0x7f012c1430c0 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f012c13c2b0 con 0x7f012c1430c0 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0124001e80 con 0x7f012c1430c0 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.472+0000 7f0128ff9640 10 monclient(hunting): dump: 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:11.474 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:11.475 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4052222120 0 0) 0x7f012c13c2b0 con 0x7f012c1430c0 2024-03-25T18:11:11.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 >> v1:172.21.15.84:6790/0 conn(0x7f012c13f710 legacy=0x7f012c1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 >> v1:172.21.15.84:6789/0 conn(0x7f012c0a98f0 legacy=0x7f012c1d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 1 monclient: found mon.b 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c137230 con 0x7f012c1430c0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f0128ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f01319cf640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f01319cf640 10 monclient: _renew_subs 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f01319cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.480+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c13b620 con 0x7f012c1430c0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.481+0000 7f01319cf640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.481+0000 7f01319cf640 10 monclient: _renew_subs 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.481+0000 7f01319cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:11.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.481+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c133710 con 0x7f012c1430c0 2024-03-25T18:11:11.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.481+0000 7f01319cf640 1 librados: init done 2024-03-25T18:11:11.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.482+0000 7f01319cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.482+0000 7f01319cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.482+0000 7f01319cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:11.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.482+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c0627b0 con 0x7f012c1430c0 2024-03-25T18:11:11.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.697+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0124001570 con 0x7f012c1430c0 2024-03-25T18:11:11.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.697+0000 7f0128ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.697+0000 7f0129ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.768+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f01240020c0 con 0x7f012c1430c0 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.768+0000 7f0128ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.768+0000 7f0128ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.768+0000 7f0128ff9640 10 monclient: dump: 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:11.769 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:11.770 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.769+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f012401ad00 con 0x7f012c1430c0 2024-03-25T18:11:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.769+0000 7f012affd640 10 monclient: get_auth_request con 0x7f010c06df00 auth_method 0 2024-03-25T18:11:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.770+0000 7f0128ff9640 10 client.4498.objecter ms_handle_connect 0x7f010c06df00 2024-03-25T18:11:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.772+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f0124054a50 con 0x7f012c1430c0 2024-03-25T18:11:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.772+0000 7f0128ff9640 10 client.4498.objecter ms_dispatch 0x7f012c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.772+0000 7f0128ff9640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.772+0000 7f0128ff9640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.772+0000 7f0128ff9640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:11:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.775+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0124050390 con 0x7f012c1430c0 2024-03-25T18:11:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.775+0000 7f0128ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:11.775+0000 7f0128ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:12.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.026+0000 7f01319cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:12.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.026+0000 7f01319cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:12.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.026+0000 7f01319cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:12.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.026+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f012c0008d0 con 0x7f012c1430c0 2024-03-25T18:11:12.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.046+0000 7f0128ff9640 1 -- 172.21.15.84:0/3995773519 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f0124052490 con 0x7f012c1430c0 2024-03-25T18:11:12.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.046+0000 7f0128ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:12.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.046+0000 7f0128ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:12.048 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:12.048 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 10 librados: watch_flush enter 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 10 librados: watch_flush exit 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 >> v1:172.21.15.157:6800/1801259026 conn(0x7f010c06df00 legacy=0x7f010c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 10 monclient: shutdown 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 >> v1:172.21.15.157:6789/0 conn(0x7f012c1430c0 legacy=0x7f012c133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 shutdown_connections 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 >> 172.21.15.84:0/3995773519 conn(0x7f012c0b7ea0 msgr2=0x7f012c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 shutdown_connections 2024-03-25T18:11:12.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 -- 172.21.15.84:0/3995773519 wait complete. 2024-03-25T18:11:12.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:12.049+0000 7f01319cf640 1 librados: shutdown 2024-03-25T18:11:12.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:18.869 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.984+0000 7f48d04c9640 10 monclient: get_monmap_and_config 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.984+0000 7f48d04c9640 10 monclient: build_initial_monmap 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.984+0000 7f48d04c9640 10 monclient: monmap: 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:18.986125+0000 2024-03-25T18:11:18.986 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:18.986125+0000 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:18.987 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:18.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.985+0000 7f48d04c9640 1 Processor -- start 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.985+0000 7f48d04c9640 1 -- start start 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.985+0000 7f48d04c9640 10 monclient: init 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-c con 0x7f48c80a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-a con 0x7f48c813f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-b con 0x7f48c81430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c8142520 con 0x7f48c813f710 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80b28a0 con 0x7f48c80a98f0 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80b2a30 con 0x7f48c81430c0 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.986+0000 7f48d04c9640 10 monclient(hunting): authenticate will time out at 7323.285809s 2024-03-25T18:11:18.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.987+0000 7f48ce265640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f48c80a98f0 0x7f48c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59168/0 (socket says 172.21.15.84:59168) 2024-03-25T18:11:18.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:18.987+0000 7f48ce265640 1 -- 172.21.15.84:0/915313622 learned_addr learned my addr 172.21.15.84:0/915313622 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:19.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670521216 0 0) 0x7f48c80b28a0 con 0x7f48c80a98f0 2024-03-25T18:11:19.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:11:19.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.004+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b4003520 con 0x7f48c80a98f0 2024-03-25T18:11:19.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442618638 0 0) 0x7f48c8142520 con 0x7f48c813f710 2024-03-25T18:11:19.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:11:19.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.014+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c80b28a0 con 0x7f48c813f710 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101588742 0 0) 0x7f48c80b2a30 con 0x7f48c81430c0 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:11:19.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.015+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c8142520 con 0x7f48c81430c0 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3319478447 0 0) 0x7f48b4003520 con 0x7f48c80a98f0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 >> v1:172.21.15.84:6790/0 conn(0x7f48c81430c0 legacy=0x7f48c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 >> v1:172.21.15.84:6789/0 conn(0x7f48c813f710 legacy=0x7f48c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: _finish_hunting 0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 monclient: found mon.noname-c 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80b44c0 con 0x7f48c80a98f0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: _finish_auth 0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: _check_auth_tickets 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f48bc001690 con 0x7f48c80a98f0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48cd263640 10 monclient: dump: 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48d04c9640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.034+0000 7f48d04c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.038+0000 7f48cd263640 1 -- 172.21.15.84:0/915313622 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48bc0017f0 con 0x7f48c80a98f0 2024-03-25T18:11:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.038+0000 7f48cd263640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.038+0000 7f48d04c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.038+0000 7f48d04c9640 4 set_mon_vals no callback set 2024-03-25T18:11:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 10 monclient: shutdown 2024-03-25T18:11:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 1 -- 172.21.15.84:0/915313622 >> v1:172.21.15.157:6789/0 conn(0x7f48c80a98f0 legacy=0x7f48c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 1 -- 172.21.15.84:0/915313622 shutdown_connections 2024-03-25T18:11:19.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 1 -- 172.21.15.84:0/915313622 >> 172.21.15.84:0/915313622 conn(0x7f48c80b7ea0 msgr2=0x7f48c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:19.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 1 -- 172.21.15.84:0/915313622 shutdown_connections 2024-03-25T18:11:19.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.039+0000 7f48d04c9640 1 -- 172.21.15.84:0/915313622 wait complete. 2024-03-25T18:11:19.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 10 monclient: build_initial_monmap 2024-03-25T18:11:19.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 10 monclient: monmap: 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 librados: starting msgr at 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 librados: starting objecter 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 Processor -- start 2024-03-25T18:11:19.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 -- start start 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 librados: setting wanted keys 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 1 librados: calling monclient init 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.040+0000 7f48d04c9640 10 monclient: init 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-c con 0x7f48c80a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-b con 0x7f48c813f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient(hunting): picked mon.noname-a con 0x7f48c81430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80aba50 con 0x7f48c81430c0 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80abd30 con 0x7f48c813f710 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80ac010 con 0x7f48c80a98f0 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:19.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.041+0000 7f48d04c9640 10 monclient(hunting): authenticate will time out at 7323.340309s 2024-03-25T18:11:19.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.053+0000 7f48ce265640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f48c80a98f0 0x7f48c81363d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57632/0 (socket says 172.21.15.84:57632) 2024-03-25T18:11:19.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.053+0000 7f48ce265640 1 -- 172.21.15.84:0/1084924921 learned_addr learned my addr 172.21.15.84:0/1084924921 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:19.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.056+0000 7f48baffd640 10 client.?.objecter ms_handle_connect 0x7f48c81430c0 2024-03-25T18:11:19.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.056+0000 7f48baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.058+0000 7f48baffd640 10 client.?.objecter ms_handle_connect 0x7f48c80a98f0 2024-03-25T18:11:19.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.058+0000 7f48baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.060+0000 7f48baffd640 10 client.?.objecter ms_handle_connect 0x7f48c813f710 2024-03-25T18:11:19.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.060+0000 7f48baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664995317 0 0) 0x7f48c80abd30 con 0x7f48c813f710 2024-03-25T18:11:19.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac004eb0 con 0x7f48c813f710 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199452769 0 0) 0x7f48c80aba50 con 0x7f48c81430c0 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:11:19.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.069+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c80abd30 con 0x7f48c81430c0 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210554461 0 0) 0x7f48c80ac010 con 0x7f48c80a98f0 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:11:19.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.074+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c80aba50 con 0x7f48c80a98f0 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478184201 0 0) 0x7f48ac004eb0 con 0x7f48c813f710 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c80ac010 con 0x7f48c813f710 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f48c4001970 con 0x7f48c813f710 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 10 monclient(hunting): dump: 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.088 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.086+0000 7f48baffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:19.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784880276 0 0) 0x7f48c80abd30 con 0x7f48c81430c0 2024-03-25T18:11:19.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac004eb0 con 0x7f48c81430c0 2024-03-25T18:11:19.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f48c0001b50 con 0x7f48c81430c0 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.091+0000 7f48baffd640 10 monclient(hunting): dump: 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:19.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 947336719 0 0) 0x7f48c80aba50 con 0x7f48c80a98f0 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c80abd30 con 0x7f48c80a98f0 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f48bc001ca0 con 0x7f48c80a98f0 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.095+0000 7f48baffd640 10 monclient(hunting): dump: 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.097 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.098 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:19.098 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:19.098 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:19.098 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:19.098 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2867426472 0 0) 0x7f48c80abd30 con 0x7f48c80a98f0 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 >> v1:172.21.15.157:6789/0 conn(0x7f48c813f710 legacy=0x7f48c80ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 >> v1:172.21.15.84:6789/0 conn(0x7f48c81430c0 legacy=0x7f48c81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 1 monclient: found mon.c 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8138fe0 con 0x7f48c80a98f0 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 5 monclient: authenticate success, global_id 4502 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 10 monclient: _renew_subs 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8137cf0 con 0x7f48c80a98f0 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 10 monclient: _renew_subs 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c8138ac0 con 0x7f48c80a98f0 2024-03-25T18:11:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.100+0000 7f48d04c9640 1 librados: init done 2024-03-25T18:11:19.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.101+0000 7f48d04c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.101+0000 7f48d04c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.101+0000 7f48d04c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:19.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.101+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c80627b0 con 0x7f48c80a98f0 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48bc004a90 con 0x7f48c80a98f0 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f48bc004da0 con 0x7f48c80a98f0 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:19.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48baffd640 10 monclient: dump: 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.393+0000 7f48cca62640 4 set_mon_vals no callback set 2024-03-25T18:11:19.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.394+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f48bc01ad70 con 0x7f48c80a98f0 2024-03-25T18:11:19.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.394+0000 7f48cda64640 10 monclient: get_auth_request con 0x7f48ac06e110 auth_method 0 2024-03-25T18:11:19.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.395+0000 7f48baffd640 10 client.4502.objecter ms_handle_connect 0x7f48ac06e110 2024-03-25T18:11:19.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f48bc0548b0 con 0x7f48c80a98f0 2024-03-25T18:11:19.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 10 client.4502.objecter ms_dispatch 0x7f48c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:19.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:19.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:19.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T18:11:19.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48bc0503c0 con 0x7f48c80a98f0 2024-03-25T18:11:19.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.399+0000 7f48baffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:19.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.620+0000 7f48d04c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.620+0000 7f48d04c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.620+0000 7f48d04c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:19.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.620+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48c8139170 con 0x7f48c80a98f0 2024-03-25T18:11:19.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.660+0000 7f48baffd640 1 -- 172.21.15.84:0/1084924921 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f48bc0522f0 con 0x7f48c80a98f0 2024-03-25T18:11:19.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.660+0000 7f48baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.660+0000 7f48baffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:19.662 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:19.662 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 10 librados: watch_flush enter 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 10 librados: watch_flush exit 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 10 client.4502.objecter successfully canceled tick 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 >> v1:172.21.15.157:6800/1801259026 conn(0x7f48ac06e110 legacy=0x7f48ac0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 10 monclient: shutdown 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 >> v1:172.21.15.84:6790/0 conn(0x7f48c80a98f0 legacy=0x7f48c81363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 shutdown_connections 2024-03-25T18:11:19.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 >> 172.21.15.84:0/1084924921 conn(0x7f48c80b7ea0 msgr2=0x7f48c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:19.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 shutdown_connections 2024-03-25T18:11:19.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 -- 172.21.15.84:0/1084924921 wait complete. 2024-03-25T18:11:19.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:19.662+0000 7f48d04c9640 1 librados: shutdown 2024-03-25T18:11:19.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:23.306 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:20.125 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.353 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:20.172 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.376 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:20.195 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.402 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:20.186 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.406 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:20.225 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.413 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:20.196 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.420 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:20.239 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.441 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:20.224 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.452 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:20.271 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:26.481 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.593+0000 7f086551e640 10 monclient: get_monmap_and_config 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.593+0000 7f086551e640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.593+0000 7f086551e640 10 monclient: monmap: 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:26.595255+0000 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:26.595255+0000 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:26.595 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:26.596 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:26.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:26.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.594+0000 7f086551e640 1 Processor -- start 2024-03-25T18:11:26.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.594+0000 7f086551e640 1 -- start start 2024-03-25T18:11:26.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.594+0000 7f086551e640 10 monclient: init 2024-03-25T18:11:26.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient(hunting): picked mon.noname-c con 0x7f08600a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient(hunting): picked mon.noname-a con 0x7f086013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient(hunting): picked mon.noname-b con 0x7f08601430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860142520 con 0x7f086013f710 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600b28a0 con 0x7f08600a98f0 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600b2a30 con 0x7f08601430c0 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.595+0000 7f086551e640 10 monclient(hunting): authenticate will time out at 7330.894714s 2024-03-25T18:11:26.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.602+0000 7f085effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f08600a98f0 0x7f08600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35646/0 (socket says 172.21.15.84:35646) 2024-03-25T18:11:26.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.603+0000 7f085effd640 1 -- 172.21.15.84:0/1543127149 learned_addr learned my addr 172.21.15.84:0/1543127149 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33502597 0 0) 0x7f08600b28a0 con 0x7f08600a98f0 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.615+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0848003520 con 0x7f08600a98f0 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338277568 0 0) 0x7f08600b2a30 con 0x7f08601430c0 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:11:26.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.623+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08600b28a0 con 0x7f08601430c0 2024-03-25T18:11:26.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790645848 0 0) 0x7f0848003520 con 0x7f08600a98f0 2024-03-25T18:11:26.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 >> v1:172.21.15.84:6790/0 conn(0x7f08601430c0 legacy=0x7f086014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 >> v1:172.21.15.84:6789/0 conn(0x7f086013f710 legacy=0x7f08601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 monclient: found mon.noname-c 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600b44c0 con 0x7f08600a98f0 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0850001a10 con 0x7f08600a98f0 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f086551e640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f086551e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f085dffb640 10 monclient: dump: 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:26.629 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.627+0000 7f086551e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0850001b70 con 0x7f08600a98f0 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 1 -- 172.21.15.84:0/1543127149 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0850001f80 con 0x7f08600a98f0 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f085dffb640 10 monclient: dump: 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f086551e640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.640+0000 7f086551e640 4 set_mon_vals no callback set 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 10 monclient: shutdown 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 1 -- 172.21.15.84:0/1543127149 >> v1:172.21.15.157:6789/0 conn(0x7f08600a98f0 legacy=0x7f08600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 1 -- 172.21.15.84:0/1543127149 shutdown_connections 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 1 -- 172.21.15.84:0/1543127149 >> 172.21.15.84:0/1543127149 conn(0x7f08600b7ea0 msgr2=0x7f08600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:26.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 1 -- 172.21.15.84:0/1543127149 shutdown_connections 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.641+0000 7f086551e640 1 -- 172.21.15.84:0/1543127149 wait complete. 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.642+0000 7f086551e640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.642+0000 7f086551e640 10 monclient: monmap: 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 librados: starting msgr at 2024-03-25T18:11:26.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 librados: starting objecter 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 Processor -- start 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 -- start start 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 librados: setting wanted keys 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 librados: calling monclient init 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient: init 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient(hunting): picked mon.noname-b con 0x7f08600a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient(hunting): picked mon.noname-c con 0x7f086013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 10 monclient(hunting): picked mon.noname-a con 0x7f08601430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860135590 con 0x7f08601430c0 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860136e50 con 0x7f08600a98f0 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.643+0000 7f086551e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860135070 con 0x7f086013f710 2024-03-25T18:11:26.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.644+0000 7f086551e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.644+0000 7f086551e640 10 monclient(hunting): authenticate will time out at 7330.942911s 2024-03-25T18:11:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.651+0000 7f085effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f08600a98f0 0x7f0860130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:35698/0 (socket says 172.21.15.84:35698) 2024-03-25T18:11:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.651+0000 7f085effd640 1 -- 172.21.15.84:0/588342266 learned_addr learned my addr 172.21.15.84:0/588342266 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.653+0000 7f08477fe640 10 client.?.objecter ms_handle_connect 0x7f08600a98f0 2024-03-25T18:11:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.653+0000 7f08477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.663+0000 7f08477fe640 10 client.?.objecter ms_handle_connect 0x7f08601430c0 2024-03-25T18:11:26.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.663+0000 7f08477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839063494 0 0) 0x7f0860136e50 con 0x7f08600a98f0 2024-03-25T18:11:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:11:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.664+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c004eb0 con 0x7f08600a98f0 2024-03-25T18:11:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.672+0000 7f08477fe640 10 client.?.objecter ms_handle_connect 0x7f086013f710 2024-03-25T18:11:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.672+0000 7f08477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042125738 0 0) 0x7f084c004eb0 con 0x7f08600a98f0 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0860136e50 con 0x7f08600a98f0 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0850001f40 con 0x7f08600a98f0 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 10 monclient(hunting): dump: 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.676+0000 7f08477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.682+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774635174 0 0) 0x7f0860135590 con 0x7f08601430c0 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.682+0000 7f08477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.682+0000 7f08477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.683+0000 7f08477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.683+0000 7f08477fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:11:26.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.683+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c004eb0 con 0x7f08601430c0 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167933553 0 0) 0x7f0860135070 con 0x7f086013f710 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:11:26.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.694+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0860135590 con 0x7f086013f710 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935764798 0 0) 0x7f0860136e50 con 0x7f08600a98f0 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 >> v1:172.21.15.84:6790/0 conn(0x7f086013f710 legacy=0x7f08601d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 >> v1:172.21.15.84:6789/0 conn(0x7f08601430c0 legacy=0x7f08601e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 1 monclient: found mon.b 2024-03-25T18:11:26.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860135d80 con 0x7f08600a98f0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f08477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 10 monclient: _renew_subs 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0860131660 con 0x7f08600a98f0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 10 monclient: _renew_subs 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08600b78a0 con 0x7f08600a98f0 2024-03-25T18:11:26.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.695+0000 7f086551e640 1 librados: init done 2024-03-25T18:11:26.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.696+0000 7f086551e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.696+0000 7f086551e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.696+0000 7f086551e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:26.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.696+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0860135f70 con 0x7f08600a98f0 2024-03-25T18:11:26.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0850001570 con 0x7f08600a98f0 2024-03-25T18:11:26.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:26.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0850004e10 con 0x7f08600a98f0 2024-03-25T18:11:26.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f08477fe640 10 monclient: dump: 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:26.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.982+0000 7f085d7fa640 4 set_mon_vals no callback set 2024-03-25T18:11:26.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.984+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f0850005030 con 0x7f08600a98f0 2024-03-25T18:11:26.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.984+0000 7f085e7fc640 10 monclient: get_auth_request con 0x7f084c06ddc0 auth_method 0 2024-03-25T18:11:26.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.984+0000 7f08477fe640 10 client.4510.objecter ms_handle_connect 0x7f084c06ddc0 2024-03-25T18:11:26.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.988+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f0850054bc0 con 0x7f08600a98f0 2024-03-25T18:11:26.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.988+0000 7f08477fe640 10 client.4510.objecter ms_dispatch 0x7f0860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.988+0000 7f08477fe640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.988+0000 7f08477fe640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.988+0000 7f08477fe640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.989+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0850050500 con 0x7f08600a98f0 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.989+0000 7f08477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:26.989+0000 7f08477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:27.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.216+0000 7f086551e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.216+0000 7f086551e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.216+0000 7f086551e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:27.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.216+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0860135200 con 0x7f08600a98f0 2024-03-25T18:11:27.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.235+0000 7f08477fe640 1 -- 172.21.15.84:0/588342266 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f0850052600 con 0x7f08600a98f0 2024-03-25T18:11:27.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.235+0000 7f08477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.235+0000 7f08477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:27.237 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:27.237 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 10 librados: watch_flush enter 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 10 librados: watch_flush exit 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 >> v1:172.21.15.157:6800/1801259026 conn(0x7f084c06ddc0 legacy=0x7f084c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 10 monclient: shutdown 2024-03-25T18:11:27.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 >> v1:172.21.15.157:6789/0 conn(0x7f08600a98f0 legacy=0x7f0860130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 shutdown_connections 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 >> 172.21.15.84:0/588342266 conn(0x7f08600b7ea0 msgr2=0x7f0860074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 shutdown_connections 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 -- 172.21.15.84:0/588342266 wait complete. 2024-03-25T18:11:27.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:27.237+0000 7f086551e640 1 librados: shutdown 2024-03-25T18:11:27.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:34.057 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.168+0000 7f13df54b640 10 monclient: get_monmap_and_config 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.168+0000 7f13df54b640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.169+0000 7f13df54b640 10 monclient: monmap: 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:34.170400+0000 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:34.170400+0000 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.169+0000 7f13df54b640 1 Processor -- start 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.169+0000 7f13df54b640 1 -- start start 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.169+0000 7f13df54b640 10 monclient: init 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-b con 0x7f13d806c380 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-c con 0x7f13d80b12d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-a con 0x7f13d80aedf0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80b16f0 con 0x7f13d80aedf0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80adfe0 con 0x7f13d80b12d0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80add00 con 0x7f13d806c380 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.170+0000 7f13df54b640 10 monclient(hunting): authenticate will time out at 7338.469491s 2024-03-25T18:11:34.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.183+0000 7f13dd2e7640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f13d806c380 0x7f13d80b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34930/0 (socket says 172.21.15.84:34930) 2024-03-25T18:11:34.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.183+0000 7f13dd2e7640 1 -- 172.21.15.84:0/1031372863 learned_addr learned my addr 172.21.15.84:0/1031372863 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:34.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450091241 0 0) 0x7f13d80add00 con 0x7f13d806c380 2024-03-25T18:11:34.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:11:34.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.205+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c4003520 con 0x7f13d806c380 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201097859 0 0) 0x7f13d80b16f0 con 0x7f13d80aedf0 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:11:34.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.211+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d80add00 con 0x7f13d80aedf0 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446552245 0 0) 0x7f13d80adfe0 con 0x7f13d80b12d0 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:11:34.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.212+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d80b16f0 con 0x7f13d80b12d0 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 96908596 0 0) 0x7f13d80add00 con 0x7f13d80aedf0 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 >> v1:172.21.15.84:6790/0 conn(0x7f13d806c380 legacy=0x7f13d80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 >> v1:172.21.15.157:6789/0 conn(0x7f13d80b12d0 legacy=0x7f13d80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 monclient: found mon.noname-a 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80ae2c0 con 0x7f13d80aedf0 2024-03-25T18:11:34.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d00019f0 con 0x7f13d80aedf0 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.238+0000 7f13cffff640 10 monclient: dump: 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:34.241 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:34.242 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.239+0000 7f13df54b640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:11:34.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.239+0000 7f13df54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:34.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d0001b50 con 0x7f13d80aedf0 2024-03-25T18:11:34.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 1 -- 172.21.15.84:0/1031372863 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d0001ee0 con 0x7f13d80aedf0 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13cffff640 10 monclient: dump: 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13df54b640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:34.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.256+0000 7f13df54b640 4 set_mon_vals no callback set 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 10 monclient: shutdown 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 1 -- 172.21.15.84:0/1031372863 >> v1:172.21.15.84:6789/0 conn(0x7f13d80aedf0 legacy=0x7f13d814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 1 -- 172.21.15.84:0/1031372863 shutdown_connections 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 1 -- 172.21.15.84:0/1031372863 >> 172.21.15.84:0/1031372863 conn(0x7f13d80b5d10 msgr2=0x7f13d80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 1 -- 172.21.15.84:0/1031372863 shutdown_connections 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.257+0000 7f13df54b640 1 -- 172.21.15.84:0/1031372863 wait complete. 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.258+0000 7f13df54b640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.258+0000 7f13df54b640 10 monclient: monmap: 2024-03-25T18:11:34.260 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.258+0000 7f13df54b640 1 librados: starting msgr at 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.258+0000 7f13df54b640 1 librados: starting objecter 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 1 Processor -- start 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 1 -- start start 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 1 librados: setting wanted keys 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 1 librados: calling monclient init 2024-03-25T18:11:34.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient: init 2024-03-25T18:11:34.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-c con 0x7f13d806c380 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-b con 0x7f13d80aedf0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.259+0000 7f13df54b640 10 monclient(hunting): picked mon.noname-a con 0x7f13d80b12d0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.260+0000 7f13df54b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d806c8e0 con 0x7f13d80b12d0 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.260+0000 7f13df54b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d81c2db0 con 0x7f13d80aedf0 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.260+0000 7f13df54b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d81c1790 con 0x7f13d806c380 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.260+0000 7f13df54b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.260+0000 7f13df54b640 10 monclient(hunting): authenticate will time out at 7338.558952s 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.268+0000 7f13ddae8640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f13d80b12d0 0x7f13d81c1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43260/0 (socket says 172.21.15.84:43260) 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.268+0000 7f13ddae8640 1 -- 172.21.15.84:0/4164224674 learned_addr learned my addr 172.21.15.84:0/4164224674 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.269+0000 7f13cdffb640 10 client.?.objecter ms_handle_connect 0x7f13d806c380 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.269+0000 7f13cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.274+0000 7f13cdffb640 10 client.?.objecter ms_handle_connect 0x7f13d80b12d0 2024-03-25T18:11:34.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.274+0000 7f13cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199740908 0 0) 0x7f13d806c8e0 con 0x7f13d80b12d0 2024-03-25T18:11:34.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:11:34.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.281+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc004eb0 con 0x7f13d80b12d0 2024-03-25T18:11:34.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.283+0000 7f13cdffb640 10 client.?.objecter ms_handle_connect 0x7f13d80aedf0 2024-03-25T18:11:34.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.284+0000 7f13cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795062035 0 0) 0x7f13d81c1790 con 0x7f13d806c380 2024-03-25T18:11:34.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:11:34.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.289+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d806c8e0 con 0x7f13d806c380 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391025465 0 0) 0x7f13d81c2db0 con 0x7f13d80aedf0 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:11:34.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.297+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d81c1790 con 0x7f13d80aedf0 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436499137 0 0) 0x7f13bc004eb0 con 0x7f13d80b12d0 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d81c2db0 con 0x7f13d80b12d0 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d00018b0 con 0x7f13d80b12d0 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 10 monclient(hunting): dump: 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:34.300 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:34.301 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:34.301 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:34.301 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.298+0000 7f13cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:34.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3243587923 0 0) 0x7f13d81c2db0 con 0x7f13d80b12d0 2024-03-25T18:11:34.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 >> v1:172.21.15.84:6790/0 conn(0x7f13d806c380 legacy=0x7f13d81c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 >> v1:172.21.15.157:6789/0 conn(0x7f13d80aedf0 legacy=0x7f13d81c0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 1 monclient: found mon.a 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d81c19d0 con 0x7f13d80b12d0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 5 monclient: authenticate success, global_id 4515 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 10 monclient: _renew_subs 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d81d8650 con 0x7f13d80b12d0 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 10 monclient: _renew_subs 2024-03-25T18:11:34.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d81c2990 con 0x7f13d80b12d0 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.307+0000 7f13df54b640 1 librados: init done 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.308+0000 7f13df54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.308+0000 7f13df54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.308+0000 7f13df54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.308+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d80b6620 con 0x7f13d80b12d0 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.311+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d0003c20 con 0x7f13d80b12d0 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.311+0000 7f13cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.311+0000 7f13cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:11:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.604+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f13d0003ff0 con 0x7f13d80b12d0 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.604+0000 7f13cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.604+0000 7f13cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.604+0000 7f13cdffb640 10 monclient: dump: 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:34.606 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:34.607 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:34.607 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.605+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f13d00041d0 con 0x7f13d80b12d0 2024-03-25T18:11:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.606+0000 7f13dd2e7640 10 monclient: get_auth_request con 0x7f13bc06dda0 auth_method 0 2024-03-25T18:11:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.606+0000 7f13cdffb640 10 client.4515.objecter ms_handle_connect 0x7f13bc06dda0 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f13d0054c80 con 0x7f13d80b12d0 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 10 client.4515.objecter ms_dispatch 0x7f13d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13d0050790 con 0x7f13d80b12d0 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.612+0000 7f13cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:34.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.835+0000 7f13df54b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.835+0000 7f13df54b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.835+0000 7f13df54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:34.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.835+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13d80627b0 con 0x7f13d80b12d0 2024-03-25T18:11:34.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.865+0000 7f13cdffb640 1 -- 172.21.15.84:0/4164224674 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f13d00526c0 con 0x7f13d80b12d0 2024-03-25T18:11:34.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.865+0000 7f13cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.865+0000 7f13cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:34.867 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:34.867 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:34.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.867+0000 7f13df54b640 10 librados: watch_flush enter 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.867+0000 7f13df54b640 10 librados: watch_flush exit 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.867+0000 7f13df54b640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.867+0000 7f13df54b640 10 client.4515.objecter successfully canceled tick 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 >> v1:172.21.15.157:6800/1801259026 conn(0x7f13bc06dda0 legacy=0x7f13bc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 10 monclient: shutdown 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 >> v1:172.21.15.84:6789/0 conn(0x7f13d80b12d0 legacy=0x7f13d81c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 shutdown_connections 2024-03-25T18:11:34.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 >> 172.21.15.84:0/4164224674 conn(0x7f13d80b5d10 msgr2=0x7f13d81cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 shutdown_connections 2024-03-25T18:11:34.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 -- 172.21.15.84:0/4164224674 wait complete. 2024-03-25T18:11:34.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:34.868+0000 7f13df54b640 1 librados: shutdown 2024-03-25T18:11:34.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:41.688 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:41.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.800+0000 7f1cdd286640 10 monclient: get_monmap_and_config 2024-03-25T18:11:41.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.800+0000 7f1cdd286640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.800+0000 7f1cdd286640 10 monclient: monmap: 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:41.802043+0000 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:41.802043+0000 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.801+0000 7f1cdd286640 1 Processor -- start 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.801+0000 7f1cdd286640 1 -- start start 2024-03-25T18:11:41.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.801+0000 7f1cdd286640 10 monclient: init 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd80a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd81430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd8142520 con 0x7f1cd80a98f0 2024-03-25T18:11:41.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd80b28a0 con 0x7f1cd81430c0 2024-03-25T18:11:41.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd80b2a30 con 0x7f1cd813f710 2024-03-25T18:11:41.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.802+0000 7f1cdd286640 10 monclient(hunting): authenticate will time out at 7346.101220s 2024-03-25T18:11:41.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.807+0000 7f1cd77fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1cd81430c0 0x7f1cd814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:33366/0 (socket says 172.21.15.84:33366) 2024-03-25T18:11:41.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.807+0000 7f1cd77fe640 1 -- 172.21.15.84:0/4082907778 learned_addr learned my addr 172.21.15.84:0/4082907778 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.821+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778170089 0 0) 0x7f1cd80b28a0 con 0x7f1cd81430c0 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.821+0000 7f1cd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.821+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.821+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.821+0000 7f1cd5ffb640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:11:41.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.822+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc0003520 con 0x7f1cd81430c0 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987963783 0 0) 0x7f1cd80b2a30 con 0x7f1cd813f710 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:11:41.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.829+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd80b28a0 con 0x7f1cd813f710 2024-03-25T18:11:41.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760956565 0 0) 0x7f1cd8142520 con 0x7f1cd80a98f0 2024-03-25T18:11:41.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:11:41.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.830+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd80b2a30 con 0x7f1cd80a98f0 2024-03-25T18:11:41.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278770996 0 0) 0x7f1cd80b28a0 con 0x7f1cd813f710 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 >> v1:172.21.15.157:6789/0 conn(0x7f1cd81430c0 legacy=0x7f1cd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 >> v1:172.21.15.84:6789/0 conn(0x7f1cd80a98f0 legacy=0x7f1cd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd80b44c0 con 0x7f1cd813f710 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cdd286640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:11:41.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.840+0000 7f1cdd286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.841+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ccc001630 con 0x7f1cd813f710 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.841+0000 7f1cd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.841+0000 7f1cd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.841+0000 7f1cd5ffb640 10 monclient: dump: 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.843 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:41.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.841+0000 7f1cdd286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ccc001880 con 0x7f1cd813f710 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 1 -- 172.21.15.84:0/4082907778 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ccc001c10 con 0x7f1cd813f710 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cd5ffb640 10 monclient: dump: 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.859 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cdd286640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:41.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.857+0000 7f1cdd286640 4 set_mon_vals no callback set 2024-03-25T18:11:41.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 10 monclient: shutdown 2024-03-25T18:11:41.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:41.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 1 -- 172.21.15.84:0/4082907778 >> v1:172.21.15.84:6790/0 conn(0x7f1cd813f710 legacy=0x7f1cd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 1 -- 172.21.15.84:0/4082907778 shutdown_connections 2024-03-25T18:11:41.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 1 -- 172.21.15.84:0/4082907778 >> 172.21.15.84:0/4082907778 conn(0x7f1cd80b7ea0 msgr2=0x7f1cd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 1 -- 172.21.15.84:0/4082907778 shutdown_connections 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.858+0000 7f1cdd286640 1 -- 172.21.15.84:0/4082907778 wait complete. 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.859+0000 7f1cdd286640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.859+0000 7f1cdd286640 10 monclient: monmap: 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.859+0000 7f1cdd286640 1 librados: starting msgr at 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.859+0000 7f1cdd286640 1 librados: starting objecter 2024-03-25T18:11:41.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.859+0000 7f1cdd286640 1 Processor -- start 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 -- start start 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 librados: setting wanted keys 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 librados: calling monclient init 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient: init 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd80a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd813f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd81430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd80b7b10 con 0x7f1cd813f710 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd813af60 con 0x7f1cd80a98f0 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd813b240 con 0x7f1cd81430c0 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.860+0000 7f1cdd286640 10 monclient(hunting): authenticate will time out at 7346.159466s 2024-03-25T18:11:41.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.877+0000 7f1cd6ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1cd80a98f0 0x7f1cd8139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:33418/0 (socket says 172.21.15.84:33418) 2024-03-25T18:11:41.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.878+0000 7f1cd6ffd640 1 -- 172.21.15.84:0/2081861927 learned_addr learned my addr 172.21.15.84:0/2081861927 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:41.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.882+0000 7f1cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1cd80a98f0 2024-03-25T18:11:41.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.882+0000 7f1cbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.883+0000 7f1cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1cd813f710 2024-03-25T18:11:41.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.883+0000 7f1cbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841014351 0 0) 0x7f1cd80b7b10 con 0x7f1cd813f710 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:11:41.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.891+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc4004eb0 con 0x7f1cd813f710 2024-03-25T18:11:41.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.893+0000 7f1cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1cd81430c0 2024-03-25T18:11:41.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.893+0000 7f1cbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99178438 0 0) 0x7f1cd813af60 con 0x7f1cd80a98f0 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.894+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd80b7b10 con 0x7f1cd80a98f0 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512199798 0 0) 0x7f1cc4004eb0 con 0x7f1cd813f710 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cd813af60 con 0x7f1cd813f710 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ccc001b90 con 0x7f1cd813f710 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.905+0000 7f1cbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379132095 0 0) 0x7f1cd813b240 con 0x7f1cd81430c0 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.910+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc4004eb0 con 0x7f1cd81430c0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4199981678 0 0) 0x7f1cd813af60 con 0x7f1cd813f710 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 >> v1:172.21.15.84:6790/0 conn(0x7f1cd81430c0 legacy=0x7f1cd81db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 >> v1:172.21.15.157:6789/0 conn(0x7f1cd80a98f0 legacy=0x7f1cd8139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 1 monclient: found mon.a 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd813b520 con 0x7f1cd813f710 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 10 monclient: _renew_subs 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd813a430 con 0x7f1cd813f710 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 10 monclient: _renew_subs 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd813b940 con 0x7f1cd813f710 2024-03-25T18:11:41.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.932+0000 7f1cdd286640 1 librados: init done 2024-03-25T18:11:41.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.933+0000 7f1cdd286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.933+0000 7f1cdd286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.933+0000 7f1cdd286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:41.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:41.933+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd80b3ef0 con 0x7f1cd813f710 2024-03-25T18:11:42.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ccc0038a0 con 0x7f1cd813f710 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ccc003bb0 con 0x7f1cd813f710 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cbf7fe640 10 monclient: dump: 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:42.237 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.235+0000 7f1cd57fa640 4 set_mon_vals no callback set 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.236+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1ccc019b80 con 0x7f1cd813f710 2024-03-25T18:11:42.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.236+0000 7f1cd6ffd640 10 monclient: get_auth_request con 0x7f1cc406de00 auth_method 0 2024-03-25T18:11:42.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.237+0000 7f1cbf7fe640 10 client.4521.objecter ms_handle_connect 0x7f1cc406de00 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.241+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1ccc0536a0 con 0x7f1cd813f710 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.241+0000 7f1cbf7fe640 10 client.4521.objecter ms_dispatch 0x7f1cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.241+0000 7f1cbf7fe640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.241+0000 7f1cbf7fe640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.242+0000 7f1cbf7fe640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.242+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ccc04f1b0 con 0x7f1cd813f710 2024-03-25T18:11:42.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.242+0000 7f1cbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:42.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.242+0000 7f1cbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:42.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.468+0000 7f1cdd286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.468+0000 7f1cdd286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.468+0000 7f1cdd286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:11:42.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.468+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cd80627b0 con 0x7f1cd813f710 2024-03-25T18:11:42.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.511+0000 7f1cbf7fe640 1 -- 172.21.15.84:0/2081861927 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1ccc0510e0 con 0x7f1cd813f710 2024-03-25T18:11:42.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.511+0000 7f1cbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.511+0000 7f1cbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:42.512 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:42.512 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 10 librados: watch_flush enter 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 10 librados: watch_flush exit 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1cc406de00 legacy=0x7f1cc40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 10 monclient: shutdown 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 >> v1:172.21.15.84:6789/0 conn(0x7f1cd813f710 legacy=0x7f1cd81dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 shutdown_connections 2024-03-25T18:11:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 >> 172.21.15.84:0/2081861927 conn(0x7f1cd80b7ea0 msgr2=0x7f1cd813ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 shutdown_connections 2024-03-25T18:11:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 -- 172.21.15.84:0/2081861927 wait complete. 2024-03-25T18:11:42.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:42.513+0000 7f1cdd286640 1 librados: shutdown 2024-03-25T18:11:42.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:43.547 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:40.365 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.555 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:40.374 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.578 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:06:40.398 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.603 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:40.387 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.612 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:40.396 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.621 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:40.441 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.628 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:40.447 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.652 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:06:40.436 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.663 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:40.483 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:49.333 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:49.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.446+0000 7f02c97b6640 10 monclient: get_monmap_and_config 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.446+0000 7f02c97b6640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.446+0000 7f02c97b6640 10 monclient: monmap: 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:49.446972+0000 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:49.446972+0000 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:49.447 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 1 Processor -- start 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 1 -- start start 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 10 monclient: init 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-c con 0x7f02c40a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.447+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-a con 0x7f02c413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-b con 0x7f02c41430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c4142520 con 0x7f02c413f710 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c40b28a0 con 0x7f02c40a98f0 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c40b2a30 con 0x7f02c41430c0 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.448+0000 7f02c97b6640 10 monclient(hunting): authenticate will time out at 7353.746033s 2024-03-25T18:11:49.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.450+0000 7f02c2ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f02c40a98f0 0x7f02c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56472/0 (socket says 172.21.15.84:56472) 2024-03-25T18:11:49.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.450+0000 7f02c2ffd640 1 -- 172.21.15.84:0/369424564 learned_addr learned my addr 172.21.15.84:0/369424564 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281742657 0 0) 0x7f02c40b28a0 con 0x7f02c40a98f0 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:11:49.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.461+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b0003520 con 0x7f02c40a98f0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346869028 0 0) 0x7f02b0003520 con 0x7f02c40a98f0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 >> v1:172.21.15.84:6790/0 conn(0x7f02c41430c0 legacy=0x7f02c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 >> v1:172.21.15.84:6789/0 conn(0x7f02c413f710 legacy=0x7f02c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c40b44c0 con 0x7f02c40a98f0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f02b4001690 con 0x7f02c40a98f0 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c97b6640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c97b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c27fc640 10 monclient: dump: 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:49.468 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.466+0000 7f02c97b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02b4001860 con 0x7f02c40a98f0 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 1 -- 172.21.15.84:0/369424564 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f02b4001cb0 con 0x7f02c40a98f0 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c97b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c97b6640 4 set_mon_vals no callback set 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.473+0000 7f02c27fc640 10 monclient: dump: 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:49.474 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 10 monclient: shutdown 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 1 -- 172.21.15.84:0/369424564 >> v1:172.21.15.157:6789/0 conn(0x7f02c40a98f0 legacy=0x7f02c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 1 -- 172.21.15.84:0/369424564 shutdown_connections 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 1 -- 172.21.15.84:0/369424564 >> 172.21.15.84:0/369424564 conn(0x7f02c40b7ea0 msgr2=0x7f02c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 1 -- 172.21.15.84:0/369424564 shutdown_connections 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.474+0000 7f02c97b6640 1 -- 172.21.15.84:0/369424564 wait complete. 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.475+0000 7f02c97b6640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.475+0000 7f02c97b6640 10 monclient: monmap: 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.475+0000 7f02c97b6640 1 librados: starting msgr at 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.475+0000 7f02c97b6640 1 librados: starting objecter 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.475+0000 7f02c97b6640 1 Processor -- start 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 1 -- start start 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 1 librados: setting wanted keys 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 1 librados: calling monclient init 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient: init 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-c con 0x7f02c40a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-b con 0x7f02c413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.476+0000 7f02c97b6640 10 monclient(hunting): picked mon.noname-a con 0x7f02c41430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.477+0000 7f02c97b6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c406fb70 con 0x7f02c41430c0 2024-03-25T18:11:49.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.477+0000 7f02c97b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c406fde0 con 0x7f02c413f710 2024-03-25T18:11:49.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.477+0000 7f02c97b6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c40700c0 con 0x7f02c40a98f0 2024-03-25T18:11:49.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.477+0000 7f02c97b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.477+0000 7f02c97b6640 10 monclient(hunting): authenticate will time out at 7353.775070s 2024-03-25T18:11:49.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.487+0000 7f02c2ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f02c40a98f0 0x7f02c4070850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48408/0 (socket says 172.21.15.84:48408) 2024-03-25T18:11:49.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.487+0000 7f02c2ffd640 1 -- 172.21.15.84:0/1900760294 learned_addr learned my addr 172.21.15.84:0/1900760294 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:49.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.487+0000 7f02bb7fe640 10 client.?.objecter ms_handle_connect 0x7f02c40a98f0 2024-03-25T18:11:49.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.487+0000 7f02bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.491+0000 7f02bb7fe640 10 client.?.objecter ms_handle_connect 0x7f02c413f710 2024-03-25T18:11:49.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.491+0000 7f02bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.497+0000 7f02bb7fe640 10 client.?.objecter ms_handle_connect 0x7f02c41430c0 2024-03-25T18:11:49.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.497+0000 7f02bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94886059 0 0) 0x7f02c40700c0 con 0x7f02c40a98f0 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:11:49.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.499+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a4004eb0 con 0x7f02c40a98f0 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711148355 0 0) 0x7f02c406fde0 con 0x7f02c413f710 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.501+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c40700c0 con 0x7f02c413f710 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016284817 0 0) 0x7f02c406fb70 con 0x7f02c41430c0 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:11:49.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.516+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c406fde0 con 0x7f02c41430c0 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995093697 0 0) 0x7f02c40700c0 con 0x7f02c413f710 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c406fb70 con 0x7f02c413f710 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f02ac0018b0 con 0x7f02c413f710 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.522+0000 7f02bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.523+0000 7f02bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594058120 0 0) 0x7f02a4004eb0 con 0x7f02c40a98f0 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c40700c0 con 0x7f02c40a98f0 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f02b4001b70 con 0x7f02c40a98f0 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.527+0000 7f02bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194735510 0 0) 0x7f02c40700c0 con 0x7f02c40a98f0 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 >> v1:172.21.15.157:6789/0 conn(0x7f02c413f710 legacy=0x7f02c406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 >> v1:172.21.15.84:6789/0 conn(0x7f02c41430c0 legacy=0x7f02c406e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 1 monclient: found mon.c 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c414e760 con 0x7f02c40a98f0 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 10 monclient: _renew_subs 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c406ecc0 con 0x7f02c40a98f0 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 10 monclient: _renew_subs 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c4071140 con 0x7f02c40a98f0 2024-03-25T18:11:49.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.532+0000 7f02c97b6640 1 librados: init done 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.533+0000 7f02c97b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.533+0000 7f02c97b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.533+0000 7f02c97b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.533+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c4139870 con 0x7f02c40a98f0 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02b4004bf0 con 0x7f02c40a98f0 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f02b4004f00 con 0x7f02c40a98f0 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02bb7fe640 10 monclient: dump: 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:49.860 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.859+0000 7f02c1ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:49.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.860+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f02b401aed0 con 0x7f02c40a98f0 2024-03-25T18:11:49.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.861+0000 7f02bbfff640 10 monclient: get_auth_request con 0x7f02a406df60 auth_method 0 2024-03-25T18:11:49.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.861+0000 7f02bb7fe640 10 client.4523.objecter ms_handle_connect 0x7f02a406df60 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f02b4054970 con 0x7f02c40a98f0 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 10 client.4523.objecter ms_dispatch 0x7f02c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02b4050480 con 0x7f02c40a98f0 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:49.879+0000 7f02bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:50.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.112+0000 7f02c97b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.112+0000 7f02c97b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.112+0000 7f02c97b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:11:50.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.112+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02c4074cc0 con 0x7f02c40a98f0 2024-03-25T18:11:50.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.145+0000 7f02bb7fe640 1 -- 172.21.15.84:0/1900760294 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f02b40523b0 con 0x7f02c40a98f0 2024-03-25T18:11:50.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.145+0000 7f02bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.145+0000 7f02bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:50.146 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:50.146 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 10 librados: watch_flush enter 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 10 librados: watch_flush exit 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 10 client.4523.objecter successfully canceled tick 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 >> v1:172.21.15.157:6800/1801259026 conn(0x7f02a406df60 legacy=0x7f02a4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 10 monclient: shutdown 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:50.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 >> v1:172.21.15.84:6790/0 conn(0x7f02c40a98f0 legacy=0x7f02c4070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 shutdown_connections 2024-03-25T18:11:50.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 >> 172.21.15.84:0/1900760294 conn(0x7f02c40b7ea0 msgr2=0x7f02c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:50.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 shutdown_connections 2024-03-25T18:11:50.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 -- 172.21.15.84:0/1900760294 wait complete. 2024-03-25T18:11:50.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:50.148+0000 7f02c97b6640 1 librados: shutdown 2024-03-25T18:11:50.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:56.965 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.077+0000 7f86f0510640 10 monclient: get_monmap_and_config 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.077+0000 7f86f0510640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.077+0000 7f86f0510640 10 monclient: monmap: 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:11:57.077985+0000 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:11:57.077985+0000 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:57.078 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:11:57.079 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:11:57.079 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.078+0000 7f86f0510640 1 Processor -- start 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.078+0000 7f86f0510640 1 -- start start 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.078+0000 7f86f0510640 10 monclient: init 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-c con 0x7f86e80b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-a con 0x7f86e80b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-b con 0x7f86e814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e80af940 con 0x7f86e80b1730 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e80b0a70 con 0x7f86e80b4d60 2024-03-25T18:11:57.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e80b0d50 con 0x7f86e814be90 2024-03-25T18:11:57.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.079+0000 7f86f0510640 10 monclient(hunting): authenticate will time out at 7361.377415s 2024-03-25T18:11:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.091+0000 7f86eeaad640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86e814be90 0x7f86e814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53258/0 (socket says 172.21.15.84:53258) 2024-03-25T18:11:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.091+0000 7f86eeaad640 1 -- 172.21.15.84:0/1488200515 learned_addr learned my addr 172.21.15.84:0/1488200515 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98675063 0 0) 0x7f86e80b0d50 con 0x7f86e814be90 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:11:57.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.099+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4003520 con 0x7f86e814be90 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584790656 0 0) 0x7f86e80af940 con 0x7f86e80b1730 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:11:57.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.109+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e80b0d50 con 0x7f86e80b1730 2024-03-25T18:11:57.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500175651 0 0) 0x7f86e80b0a70 con 0x7f86e80b4d60 2024-03-25T18:11:57.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:11:57.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.119+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e80af940 con 0x7f86e80b4d60 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4194190722 0 0) 0x7f86e80b0d50 con 0x7f86e80b1730 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 >> v1:172.21.15.84:6790/0 conn(0x7f86e814be90 legacy=0x7f86e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 >> v1:172.21.15.157:6789/0 conn(0x7f86e80b4d60 legacy=0x7f86e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 monclient: found mon.noname-a 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e80b1030 con 0x7f86e80b1730 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86d8001670 con 0x7f86e80b1730 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86ed2aa640 10 monclient: dump: 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86f0510640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.130+0000 7f86f0510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:57.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d80017d0 con 0x7f86e80b1730 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 1 -- 172.21.15.84:0/1488200515 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86d8001be0 con 0x7f86e80b1730 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86ed2aa640 10 monclient: dump: 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.145 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.146 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.146 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86f0510640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:57.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.144+0000 7f86f0510640 4 set_mon_vals no callback set 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 10 monclient: shutdown 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 1 -- 172.21.15.84:0/1488200515 >> v1:172.21.15.84:6789/0 conn(0x7f86e80b1730 legacy=0x7f86e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 1 -- 172.21.15.84:0/1488200515 shutdown_connections 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 1 -- 172.21.15.84:0/1488200515 >> 172.21.15.84:0/1488200515 conn(0x7f86e80617a0 msgr2=0x7f86e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 1 -- 172.21.15.84:0/1488200515 shutdown_connections 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.145+0000 7f86f0510640 1 -- 172.21.15.84:0/1488200515 wait complete. 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.146+0000 7f86f0510640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.146+0000 7f86f0510640 10 monclient: monmap: 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.146+0000 7f86f0510640 1 librados: starting msgr at 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.146+0000 7f86f0510640 1 librados: starting objecter 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.146+0000 7f86f0510640 1 Processor -- start 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 -- start start 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 librados: setting wanted keys 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 librados: calling monclient init 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient: init 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-a con 0x7f86e80b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-c con 0x7f86e80b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient(hunting): picked mon.noname-b con 0x7f86e814be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e813aa80 con 0x7f86e80b1730 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e8137c60 con 0x7f86e814be90 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e81326c0 con 0x7f86e80b4d60 2024-03-25T18:11:57.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.147+0000 7f86f0510640 10 monclient(hunting): authenticate will time out at 7361.445867s 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.167+0000 7f86ee2ac640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f86e80b1730 0x7f86e8131fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44428/0 (socket says 172.21.15.84:44428) 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.167+0000 7f86ee2ac640 1 -- 172.21.15.84:0/1709628714 learned_addr learned my addr 172.21.15.84:0/1709628714 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.169+0000 7f86deffd640 10 client.?.objecter ms_handle_connect 0x7f86e80b1730 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.169+0000 7f86deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.174+0000 7f86deffd640 10 client.?.objecter ms_handle_connect 0x7f86e814be90 2024-03-25T18:11:57.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.174+0000 7f86deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.175+0000 7f86deffd640 10 client.?.objecter ms_handle_connect 0x7f86e80b4d60 2024-03-25T18:11:57.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.175+0000 7f86deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730062126 0 0) 0x7f86e813aa80 con 0x7f86e80b1730 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:11:57.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.184+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86cc004eb0 con 0x7f86e80b1730 2024-03-25T18:11:57.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613053578 0 0) 0x7f86e81326c0 con 0x7f86e80b4d60 2024-03-25T18:11:57.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:11:57.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.188+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e813aa80 con 0x7f86e80b4d60 2024-03-25T18:11:57.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378046435 0 0) 0x7f86e8137c60 con 0x7f86e814be90 2024-03-25T18:11:57.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:11:57.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.189+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e81326c0 con 0x7f86e814be90 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 631253805 0 0) 0x7f86cc004eb0 con 0x7f86e80b1730 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e8137c60 con 0x7f86e80b1730 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86e4001a10 con 0x7f86e80b1730 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 10 monclient(hunting): dump: 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.204 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.204+0000 7f86deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:57.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4142640326 0 0) 0x7f86e81326c0 con 0x7f86e814be90 2024-03-25T18:11:57.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86cc004eb0 con 0x7f86e814be90 2024-03-25T18:11:57.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86e0001a30 con 0x7f86e814be90 2024-03-25T18:11:57.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.212+0000 7f86deffd640 10 monclient(hunting): dump: 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.213 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321359808 0 0) 0x7f86e813aa80 con 0x7f86e80b4d60 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e81326c0 con 0x7f86e80b4d60 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86d8001ae0 con 0x7f86e80b4d60 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.224+0000 7f86deffd640 10 monclient(hunting): dump: 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.225 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.226 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.226 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.226 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1287414376 0 0) 0x7f86cc004eb0 con 0x7f86e814be90 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 >> v1:172.21.15.84:6790/0 conn(0x7f86e80b4d60 legacy=0x7f86e81d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 >> v1:172.21.15.84:6789/0 conn(0x7f86e80b1730 legacy=0x7f86e8131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 1 monclient: found mon.b 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e8138440 con 0x7f86e814be90 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 10 monclient: _renew_subs 2024-03-25T18:11:57.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e8138e60 con 0x7f86e814be90 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 10 monclient: _renew_subs 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e8145260 con 0x7f86e814be90 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.232+0000 7f86f0510640 1 librados: init done 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.233+0000 7f86f0510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.233+0000 7f86f0510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.233+0000 7f86f0510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:57.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.233+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e80627b0 con 0x7f86e814be90 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86e0001f20 con 0x7f86e814be90 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f86e0004d00 con 0x7f86e814be90 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86deffd640 10 monclient: dump: 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.525 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.524+0000 7f86ecaa9640 4 set_mon_vals no callback set 2024-03-25T18:11:57.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.525+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f86e001acd0 con 0x7f86e814be90 2024-03-25T18:11:57.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.526+0000 7f86ee2ac640 10 monclient: get_auth_request con 0x7f86cc06e110 auth_method 0 2024-03-25T18:11:57.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.526+0000 7f86deffd640 10 client.4534.objecter ms_handle_connect 0x7f86cc06e110 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.529+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f86e0054860 con 0x7f86e814be90 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.529+0000 7f86deffd640 10 client.4534.objecter ms_dispatch 0x7f86e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.529+0000 7f86deffd640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.529+0000 7f86deffd640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.529+0000 7f86deffd640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.530+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86e0050370 con 0x7f86e814be90 2024-03-25T18:11:57.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.530+0000 7f86deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.530+0000 7f86deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:57.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.788+0000 7f86f0510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.788+0000 7f86f0510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.788+0000 7f86f0510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:11:57.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.788+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86e81385d0 con 0x7f86e814be90 2024-03-25T18:11:57.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.837+0000 7f86deffd640 1 -- 172.21.15.84:0/1709628714 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f86e00522a0 con 0x7f86e814be90 2024-03-25T18:11:57.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.837+0000 7f86deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.837+0000 7f86deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:57.838 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:11:57.838 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:57.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 10 librados: watch_flush enter 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 10 librados: watch_flush exit 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 >> v1:172.21.15.157:6800/1801259026 conn(0x7f86cc06e110 legacy=0x7f86cc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 10 monclient: shutdown 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 >> v1:172.21.15.157:6789/0 conn(0x7f86e814be90 legacy=0x7f86e81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 shutdown_connections 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 >> 172.21.15.84:0/1709628714 conn(0x7f86e80617a0 msgr2=0x7f86e813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 shutdown_connections 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 -- 172.21.15.84:0/1709628714 wait complete. 2024-03-25T18:11:57.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:11:57.840+0000 7f86f0510640 1 librados: shutdown 2024-03-25T18:11:57.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:03.476 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:00.295 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.484 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:00.303 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.510 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:00.329 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.525 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:00.308 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.534 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:00.318 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.551 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:00.370 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.557 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:00.376 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.558 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:00.342 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.585 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:00.404 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:04.658 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.771+0000 7efe788d4640 10 monclient: get_monmap_and_config 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.771+0000 7efe788d4640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.771+0000 7efe788d4640 10 monclient: monmap: 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:04.772130+0000 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:04.772130+0000 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:04.772 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:04.773 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:04.773 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.772+0000 7efe788d4640 1 Processor -- start 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.772+0000 7efe788d4640 1 -- start start 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.772+0000 7efe788d4640 10 monclient: init 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-a con 0x7efe740b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-b con 0x7efe740b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-c con 0x7efe7414be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe740af940 con 0x7efe740b4d60 2024-03-25T18:12:04.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe740b0a70 con 0x7efe7414be90 2024-03-25T18:12:04.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe740b0d50 con 0x7efe740b1730 2024-03-25T18:12:04.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.773+0000 7efe788d4640 10 monclient(hunting): authenticate will time out at 7369.071419s 2024-03-25T18:12:04.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.776+0000 7efe72d9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7efe7414be90 0x7efe7414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:47346/0 (socket says 172.21.15.84:47346) 2024-03-25T18:12:04.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.776+0000 7efe72d9d640 1 -- 172.21.15.84:0/3634229640 learned_addr learned my addr 172.21.15.84:0/3634229640 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:04.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447451232 0 0) 0x7efe740b0a70 con 0x7efe7414be90 2024-03-25T18:12:04.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:12:04.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.785+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c003520 con 0x7efe7414be90 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039109707 0 0) 0x7efe5c003520 con 0x7efe7414be90 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 >> v1:172.21.15.84:6790/0 conn(0x7efe740b1730 legacy=0x7efe740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 >> v1:172.21.15.84:6789/0 conn(0x7efe740b4d60 legacy=0x7efe7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 monclient: found mon.noname-c 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740b1030 con 0x7efe7414be90 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: _finish_auth 0 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efe6c001a10 con 0x7efe7414be90 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe788d4640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:12:04.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe7159a640 10 monclient: dump: 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:04.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.794+0000 7efe788d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:04.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe6c001b70 con 0x7efe7414be90 2024-03-25T18:12:04.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:04.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:04.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 4 set_mon_vals no callback set 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 1 -- 172.21.15.84:0/3634229640 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efe6c001f80 con 0x7efe7414be90 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe7159a640 10 monclient: dump: 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:04.802 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 10 monclient: shutdown 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 1 -- 172.21.15.84:0/3634229640 >> v1:172.21.15.157:6789/0 conn(0x7efe7414be90 legacy=0x7efe7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.801+0000 7efe788d4640 1 -- 172.21.15.84:0/3634229640 shutdown_connections 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.802+0000 7efe788d4640 1 -- 172.21.15.84:0/3634229640 >> 172.21.15.84:0/3634229640 conn(0x7efe740617a0 msgr2=0x7efe74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.802+0000 7efe788d4640 1 -- 172.21.15.84:0/3634229640 shutdown_connections 2024-03-25T18:12:04.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.802+0000 7efe788d4640 1 -- 172.21.15.84:0/3634229640 wait complete. 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.802+0000 7efe788d4640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.802+0000 7efe788d4640 10 monclient: monmap: 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:04.805 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 librados: starting msgr at 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 librados: starting objecter 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 Processor -- start 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 -- start start 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 librados: setting wanted keys 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 1 librados: calling monclient init 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.803+0000 7efe788d4640 10 monclient: init 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-b con 0x7efe740b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-a con 0x7efe740b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient(hunting): picked mon.noname-c con 0x7efe7414be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe74143900 con 0x7efe740b4d60 2024-03-25T18:12:04.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe74136070 con 0x7efe740b1730 2024-03-25T18:12:04.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe741dce50 con 0x7efe7414be90 2024-03-25T18:12:04.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.804+0000 7efe788d4640 10 monclient(hunting): authenticate will time out at 7369.102065s 2024-03-25T18:12:04.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.809+0000 7efe7259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7efe740b1730 0x7efe74137e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:47368/0 (socket says 172.21.15.84:47368) 2024-03-25T18:12:04.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.809+0000 7efe7259c640 1 -- 172.21.15.84:0/4285681038 learned_addr learned my addr 172.21.15.84:0/4285681038 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:04.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.812+0000 7efe62ffd640 10 client.?.objecter ms_handle_connect 0x7efe740b1730 2024-03-25T18:12:04.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.812+0000 7efe62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.821+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584155539 0 0) 0x7efe74136070 con 0x7efe740b1730 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.822+0000 7efe62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.822+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.822+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.822+0000 7efe62ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:12:04.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.822+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe54004eb0 con 0x7efe740b1730 2024-03-25T18:12:04.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.824+0000 7efe62ffd640 10 client.?.objecter ms_handle_connect 0x7efe740b4d60 2024-03-25T18:12:04.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.824+0000 7efe62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.832+0000 7efe62ffd640 10 client.?.objecter ms_handle_connect 0x7efe7414be90 2024-03-25T18:12:04.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.832+0000 7efe62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904822201 0 0) 0x7efe74143900 con 0x7efe740b4d60 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:12:04.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.836+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe74136070 con 0x7efe740b4d60 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124356197 0 0) 0x7efe741dce50 con 0x7efe7414be90 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:12:04.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.837+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe74143900 con 0x7efe7414be90 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096820868 0 0) 0x7efe54004eb0 con 0x7efe740b1730 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe741dce50 con 0x7efe740b1730 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efe64001a30 con 0x7efe740b1730 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:04.840 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:04.841 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:04.841 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:04.841 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:04.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.839+0000 7efe62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1934130700 0 0) 0x7efe741dce50 con 0x7efe740b1730 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 >> v1:172.21.15.84:6790/0 conn(0x7efe7414be90 legacy=0x7efe741dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 >> v1:172.21.15.84:6789/0 conn(0x7efe740b4d60 legacy=0x7efe741dc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 1 monclient: found mon.b 2024-03-25T18:12:04.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe741374e0 con 0x7efe740b1730 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe788d4640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe788d4640 10 monclient: _renew_subs 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe788d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.848+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe741377c0 con 0x7efe740b1730 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 monclient: _renew_subs 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe74135b00 con 0x7efe740b1730 2024-03-25T18:12:04.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 1 librados: init done 2024-03-25T18:12:04.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:04.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:04.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:04.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:04.849+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe74135e00 con 0x7efe740b1730 2024-03-25T18:12:05.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.131+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe64001c70 con 0x7efe740b1730 2024-03-25T18:12:05.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:05.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efe64003a60 con 0x7efe740b1730 2024-03-25T18:12:05.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe62ffd640 10 monclient: dump: 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:05.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.132+0000 7efe70d99640 4 set_mon_vals no callback set 2024-03-25T18:12:05.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.133+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7efe64003c40 con 0x7efe740b1730 2024-03-25T18:12:05.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.133+0000 7efe71d9b640 10 monclient: get_auth_request con 0x7efe5406de20 auth_method 0 2024-03-25T18:12:05.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.134+0000 7efe62ffd640 10 client.4540.objecter ms_handle_connect 0x7efe5406de20 2024-03-25T18:12:05.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7efe64054850 con 0x7efe740b1730 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 10 client.4540.objecter ms_dispatch 0x7efe74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe64050360 con 0x7efe740b1730 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:05.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.138+0000 7efe62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:05.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.393+0000 7efe788d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.393+0000 7efe788d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.393+0000 7efe788d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:05.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.393+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe740627b0 con 0x7efe740b1730 2024-03-25T18:12:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.417+0000 7efe62ffd640 1 -- 172.21.15.84:0/4285681038 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7efe64052290 con 0x7efe740b1730 2024-03-25T18:12:05.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.417+0000 7efe62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.417+0000 7efe62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:05.418 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:05.418 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 10 librados: watch_flush enter 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 10 librados: watch_flush exit 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 >> v1:172.21.15.157:6800/1801259026 conn(0x7efe5406de20 legacy=0x7efe540702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 10 monclient: shutdown 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 >> v1:172.21.15.157:6789/0 conn(0x7efe740b1730 legacy=0x7efe74137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 shutdown_connections 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 >> 172.21.15.84:0/4285681038 conn(0x7efe740617a0 msgr2=0x7efe7413c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:05.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 shutdown_connections 2024-03-25T18:12:05.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 -- 172.21.15.84:0/4285681038 wait complete. 2024-03-25T18:12:05.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:05.420+0000 7efe788d4640 1 librados: shutdown 2024-03-25T18:12:05.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:12.239 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.352+0000 7ffac7c82640 10 monclient: get_monmap_and_config 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.352+0000 7ffac7c82640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.352+0000 7ffac7c82640 10 monclient: monmap: 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:12.354031+0000 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:12.354031+0000 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:12.354 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:12.355 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:12.355 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.353+0000 7ffac7c82640 1 Processor -- start 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.353+0000 7ffac7c82640 1 -- start start 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.353+0000 7ffac7c82640 10 monclient: init 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-a con 0x7ffac00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-c con 0x7ffac013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-b con 0x7ffac01430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:12.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0142520 con 0x7ffac00a98f0 2024-03-25T18:12:12.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00b28a0 con 0x7ffac013f710 2024-03-25T18:12:12.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00b2a30 con 0x7ffac01430c0 2024-03-25T18:12:12.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.354+0000 7ffac7c82640 10 monclient(hunting): authenticate will time out at 7376.653669s 2024-03-25T18:12:12.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.358+0000 7ffac5a1e640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ffac00a98f0 0x7ffac00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35118/0 (socket says 172.21.15.84:35118) 2024-03-25T18:12:12.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.358+0000 7ffac5a1e640 1 -- 172.21.15.84:0/4070572161 learned_addr learned my addr 172.21.15.84:0/4070572161 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781536995 0 0) 0x7ffac0142520 con 0x7ffac00a98f0 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:12:12.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.365+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac003520 con 0x7ffac00a98f0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635424015 0 0) 0x7ffaac003520 con 0x7ffac00a98f0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 >> v1:172.21.15.84:6790/0 conn(0x7ffac01430c0 legacy=0x7ffac014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 >> v1:172.21.15.157:6789/0 conn(0x7ffac013f710 legacy=0x7ffac01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 monclient: found mon.noname-a 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00b44c0 con 0x7ffac00a98f0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffabc001690 con 0x7ffac00a98f0 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac7c82640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac7c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac4a1c640 10 monclient: dump: 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:12.381 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.382 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.378+0000 7ffac7c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffabc0017f0 con 0x7ffac00a98f0 2024-03-25T18:12:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 1 -- 172.21.15.84:0/4070572161 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffabc001c00 con 0x7ffac00a98f0 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac7c82640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac7c82640 4 set_mon_vals no callback set 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.386+0000 7ffac4a1c640 10 monclient: dump: 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:12.388 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 10 monclient: shutdown 2024-03-25T18:12:12.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 1 -- 172.21.15.84:0/4070572161 >> v1:172.21.15.84:6789/0 conn(0x7ffac00a98f0 legacy=0x7ffac00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 1 -- 172.21.15.84:0/4070572161 shutdown_connections 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 1 -- 172.21.15.84:0/4070572161 >> 172.21.15.84:0/4070572161 conn(0x7ffac00b7ea0 msgr2=0x7ffac00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 1 -- 172.21.15.84:0/4070572161 shutdown_connections 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 1 -- 172.21.15.84:0/4070572161 wait complete. 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.387+0000 7ffac7c82640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient: monmap: 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:12.391 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 librados: starting msgr at 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 librados: starting objecter 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 Processor -- start 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 -- start start 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 librados: setting wanted keys 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 1 librados: calling monclient init 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient: init 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-b con 0x7ffac00a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.388+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-a con 0x7ffac013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 10 monclient(hunting): picked mon.noname-c con 0x7ffac01430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00abd30 con 0x7ffac013f710 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac01d77a0 con 0x7ffac00a98f0 2024-03-25T18:12:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac01d51f0 con 0x7ffac01430c0 2024-03-25T18:12:12.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.389+0000 7ffac7c82640 10 monclient(hunting): authenticate will time out at 7376.688004s 2024-03-25T18:12:12.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.400+0000 7ffac621f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ffac01430c0 0x7ffac01e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33598/0 (socket says 172.21.15.84:33598) 2024-03-25T18:12:12.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.400+0000 7ffac621f640 1 -- 172.21.15.84:0/4217060247 learned_addr learned my addr 172.21.15.84:0/4217060247 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:12.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.401+0000 7ffab67fc640 10 client.?.objecter ms_handle_connect 0x7ffac00a98f0 2024-03-25T18:12:12.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.401+0000 7ffab67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.402+0000 7ffab67fc640 10 client.?.objecter ms_handle_connect 0x7ffac01430c0 2024-03-25T18:12:12.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.402+0000 7ffab67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87741054 0 0) 0x7ffac01d51f0 con 0x7ffac01430c0 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:12:12.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.408+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4004eb0 con 0x7ffac01430c0 2024-03-25T18:12:12.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.412+0000 7ffab67fc640 10 client.?.objecter ms_handle_connect 0x7ffac013f710 2024-03-25T18:12:12.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.412+0000 7ffab67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243710394 0 0) 0x7ffaa4004eb0 con 0x7ffac01430c0 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac01d51f0 con 0x7ffac01430c0 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffab8001c30 con 0x7ffac01430c0 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 10 monclient(hunting): dump: 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.419 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.417+0000 7ffab67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234429326 0 0) 0x7ffac01d77a0 con 0x7ffac00a98f0 2024-03-25T18:12:12.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:12:12.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.419+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4004eb0 con 0x7ffac00a98f0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3110214500 0 0) 0x7ffac01d51f0 con 0x7ffac01430c0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 >> v1:172.21.15.157:6789/0 conn(0x7ffac00a98f0 legacy=0x7ffac01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 >> v1:172.21.15.84:6789/0 conn(0x7ffac013f710 legacy=0x7ffac0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 1 monclient: found mon.c 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac01d83d0 con 0x7ffac01430c0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:12:12.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 10 monclient: _renew_subs 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac0138a50 con 0x7ffac01430c0 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 10 monclient: _renew_subs 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac01e35c0 con 0x7ffac01430c0 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.426+0000 7ffac7c82640 1 librados: init done 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.427+0000 7ffac7c82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.427+0000 7ffac7c82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.427+0000 7ffac7c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:12.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.427+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac00627b0 con 0x7ffac01430c0 2024-03-25T18:12:12.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab8003990 con 0x7ffac01430c0 2024-03-25T18:12:12.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ffab8003de0 con 0x7ffac01430c0 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab67fc640 10 monclient: dump: 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:12.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:12.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.736+0000 7ffab7fff640 4 set_mon_vals no callback set 2024-03-25T18:12:12.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.737+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7ffab8004000 con 0x7ffac01430c0 2024-03-25T18:12:12.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.738+0000 7ffac5a1e640 10 monclient: get_auth_request con 0x7ffaa406dd60 auth_method 0 2024-03-25T18:12:12.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.738+0000 7ffab67fc640 10 client.4538.objecter ms_handle_connect 0x7ffaa406dd60 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7ffab8054ad0 con 0x7ffac01430c0 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 10 client.4538.objecter ms_dispatch 0x7ffac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffab80505e0 con 0x7ffac01430c0 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.742+0000 7ffab67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:12.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.979+0000 7ffac7c82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:12.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.979+0000 7ffac7c82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:12.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.979+0000 7ffac7c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:12.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:12.979+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffac01d8560 con 0x7ffac01430c0 2024-03-25T18:12:13.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.008+0000 7ffab67fc640 1 -- 172.21.15.84:0/4217060247 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7ffab8052510 con 0x7ffac01430c0 2024-03-25T18:12:13.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.008+0000 7ffab67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.008+0000 7ffab67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:13.010 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:13.010 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:13.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.010+0000 7ffac7c82640 10 librados: watch_flush enter 2024-03-25T18:12:13.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.010+0000 7ffac7c82640 10 librados: watch_flush exit 2024-03-25T18:12:13.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.010+0000 7ffac7c82640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:12:13.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.010+0000 7ffac7c82640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 >> v1:172.21.15.157:6800/1801259026 conn(0x7ffaa406dd60 legacy=0x7ffaa4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 10 monclient: shutdown 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 >> v1:172.21.15.84:6790/0 conn(0x7ffac01430c0 legacy=0x7ffac01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 shutdown_connections 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 >> 172.21.15.84:0/4217060247 conn(0x7ffac00b7ea0 msgr2=0x7ffac0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 shutdown_connections 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 -- 172.21.15.84:0/4217060247 wait complete. 2024-03-25T18:12:13.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:13.011+0000 7ffac7c82640 1 librados: shutdown 2024-03-25T18:12:13.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:19.829 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:19.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.943+0000 7f7936d15640 10 monclient: get_monmap_and_config 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.943+0000 7f7936d15640 10 monclient: build_initial_monmap 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.943+0000 7f7936d15640 10 monclient: monmap: 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:19.944653+0000 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:19.944653+0000 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:19.945 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:19.946 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:19.946 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.944+0000 7f7936d15640 1 Processor -- start 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.944+0000 7f7936d15640 1 -- start start 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.944+0000 7f7936d15640 10 monclient: init 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-a con 0x7f79300a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-c con 0x7f793013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-b con 0x7f79301430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7930142520 con 0x7f79300a98f0 2024-03-25T18:12:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79300b28a0 con 0x7f793013f710 2024-03-25T18:12:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79300b2a30 con 0x7f79301430c0 2024-03-25T18:12:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.945+0000 7f7936d15640 10 monclient(hunting): authenticate will time out at 7384.244248s 2024-03-25T18:12:19.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.951+0000 7f7934ab1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f79300a98f0 0x7f79300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52292/0 (socket says 172.21.15.84:52292) 2024-03-25T18:12:19.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.951+0000 7f7934ab1640 1 -- 172.21.15.84:0/2308001312 learned_addr learned my addr 172.21.15.84:0/2308001312 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223963 0 0) 0x7f79300b28a0 con 0x7f793013f710 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:12:19.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.963+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7920003520 con 0x7f793013f710 2024-03-25T18:12:19.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792411831 0 0) 0x7f79300b2a30 con 0x7f79301430c0 2024-03-25T18:12:19.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79300b28a0 con 0x7f79301430c0 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994500708 0 0) 0x7f7930142520 con 0x7f79300a98f0 2024-03-25T18:12:19.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:19.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:19.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:19.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:12:19.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.969+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79300b2a30 con 0x7f79300a98f0 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293218329 0 0) 0x7f7920003520 con 0x7f793013f710 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 >> v1:172.21.15.84:6790/0 conn(0x7f79301430c0 legacy=0x7f793014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 >> v1:172.21.15.84:6789/0 conn(0x7f79300a98f0 legacy=0x7f79300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:19.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79300b44c0 con 0x7f793013f710 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f791c001690 con 0x7f793013f710 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f7936d15640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f7936d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f792f7fe640 10 monclient: dump: 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:19.974 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:19.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.971+0000 7f7936d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.980+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f791c0017f0 con 0x7f793013f710 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.980+0000 7f792f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f792f7fe640 1 -- 172.21.15.84:0/2308001312 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f791c001c00 con 0x7f793013f710 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 4 set_mon_vals no callback set 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f792f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:19.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f792f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f792f7fe640 10 monclient: dump: 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:19.983 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 10 monclient: shutdown 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 1 -- 172.21.15.84:0/2308001312 >> v1:172.21.15.157:6789/0 conn(0x7f793013f710 legacy=0x7f79301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 1 -- 172.21.15.84:0/2308001312 shutdown_connections 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.981+0000 7f7936d15640 1 -- 172.21.15.84:0/2308001312 >> 172.21.15.84:0/2308001312 conn(0x7f79300b7ea0 msgr2=0x7f79300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.982+0000 7f7936d15640 1 -- 172.21.15.84:0/2308001312 shutdown_connections 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.982+0000 7f7936d15640 1 -- 172.21.15.84:0/2308001312 wait complete. 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.982+0000 7f7936d15640 10 monclient: build_initial_monmap 2024-03-25T18:12:19.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.982+0000 7f7936d15640 10 monclient: monmap: 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 librados: starting msgr at 2024-03-25T18:12:19.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 librados: starting objecter 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 Processor -- start 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 -- start start 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 librados: setting wanted keys 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 1 librados: calling monclient init 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.983+0000 7f7936d15640 10 monclient: init 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-b con 0x7f79300a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-c con 0x7f793013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient(hunting): picked mon.noname-a con 0x7f79301430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7930138a50 con 0x7f79301430c0 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79300ab9e0 con 0x7f79300a98f0 2024-03-25T18:12:19.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79301d7880 con 0x7f793013f710 2024-03-25T18:12:19.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:19.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.984+0000 7f7936d15640 10 monclient(hunting): authenticate will time out at 7384.283401s 2024-03-25T18:12:19.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.988+0000 7f792ffff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f793013f710 0x7f79301d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58598/0 (socket says 172.21.15.84:58598) 2024-03-25T18:12:19.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.988+0000 7f792ffff640 1 -- 172.21.15.84:0/2057070554 learned_addr learned my addr 172.21.15.84:0/2057070554 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:19.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.990+0000 7f792d7fa640 10 client.?.objecter ms_handle_connect 0x7f793013f710 2024-03-25T18:12:19.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:19.991+0000 7f792d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.000+0000 7f792d7fa640 10 client.?.objecter ms_handle_connect 0x7f79300a98f0 2024-03-25T18:12:20.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.000+0000 7f792d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.006+0000 7f792d7fa640 10 client.?.objecter ms_handle_connect 0x7f79301430c0 2024-03-25T18:12:20.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.006+0000 7f792d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713008880 0 0) 0x7f79301d7880 con 0x7f793013f710 2024-03-25T18:12:20.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:12:20.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.014+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7914004eb0 con 0x7f793013f710 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303583769 0 0) 0x7f79300ab9e0 con 0x7f79300a98f0 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:12:20.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.016+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79301d7880 con 0x7f79300a98f0 2024-03-25T18:12:20.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107073066 0 0) 0x7f7930138a50 con 0x7f79301430c0 2024-03-25T18:12:20.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:12:20.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.024+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79300ab9e0 con 0x7f79301430c0 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1889542167 0 0) 0x7f79301d7880 con 0x7f79300a98f0 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7930138a50 con 0x7f79300a98f0 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f7924001870 con 0x7f79300a98f0 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): dump: 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:20.051 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473779013 0 0) 0x7f79300ab9e0 con 0x7f79301430c0 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79301d7880 con 0x7f79301430c0 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f7928001b50 con 0x7f79301430c0 2024-03-25T18:12:20.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.049+0000 7f792d7fa640 10 monclient(hunting): dump: 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:20.053 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:20.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3866805687 0 0) 0x7f7914004eb0 con 0x7f793013f710 2024-03-25T18:12:20.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79300ab9e0 con 0x7f793013f710 2024-03-25T18:12:20.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f791c001bc0 con 0x7f793013f710 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.067+0000 7f792d7fa640 10 monclient(hunting): dump: 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:20.069 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3949015617 0 0) 0x7f79301d7880 con 0x7f79301430c0 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 >> v1:172.21.15.84:6790/0 conn(0x7f793013f710 legacy=0x7f79301d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 >> v1:172.21.15.157:6789/0 conn(0x7f79300a98f0 legacy=0x7f7930138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 1 monclient: found mon.a 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79301d84b0 con 0x7f79301430c0 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:20.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f792d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 10 monclient: _renew_subs 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79301d51f0 con 0x7f79301430c0 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 10 monclient: _renew_subs 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7930138e60 con 0x7f79301430c0 2024-03-25T18:12:20.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.070+0000 7f7936d15640 1 librados: init done 2024-03-25T18:12:20.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.071+0000 7f7936d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.071+0000 7f7936d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.071+0000 7f7936d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:20.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.071+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79301d76e0 con 0x7f79301430c0 2024-03-25T18:12:20.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7928001fa0 con 0x7f79301430c0 2024-03-25T18:12:20.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f7928003970 con 0x7f79301430c0 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792d7fa640 10 monclient: dump: 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:20.393 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:20.394 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:20.394 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:20.394 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:20.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.391+0000 7f792effd640 4 set_mon_vals no callback set 2024-03-25T18:12:20.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.392+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f7928019940 con 0x7f79301430c0 2024-03-25T18:12:20.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.392+0000 7f7934ab1640 10 monclient: get_auth_request con 0x7f791406e080 auth_method 0 2024-03-25T18:12:20.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.393+0000 7f792d7fa640 10 client.4551.objecter ms_handle_connect 0x7f791406e080 2024-03-25T18:12:20.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f7928053620 con 0x7f79301430c0 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 10 client.4551.objecter ms_dispatch 0x7f7930000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f792804ef60 con 0x7f79301430c0 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.399+0000 7f792d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.615+0000 7f7936d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.615+0000 7f7936d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.615+0000 7f7936d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.615+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79301d7a10 con 0x7f79301430c0 2024-03-25T18:12:20.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.650+0000 7f792d7fa640 1 -- 172.21.15.84:0/2057070554 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f7928051060 con 0x7f79301430c0 2024-03-25T18:12:20.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.650+0000 7f792d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.650+0000 7f792d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:20.652 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:20.652 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 10 librados: watch_flush enter 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 10 librados: watch_flush exit 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 10 client.4551.objecter successfully canceled tick 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 >> v1:172.21.15.157:6800/1801259026 conn(0x7f791406e080 legacy=0x7f7914070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 10 monclient: shutdown 2024-03-25T18:12:20.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 >> v1:172.21.15.84:6789/0 conn(0x7f79301430c0 legacy=0x7f79301e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 shutdown_connections 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 >> 172.21.15.84:0/2057070554 conn(0x7f79300b7ea0 msgr2=0x7f7930074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 shutdown_connections 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 -- 172.21.15.84:0/2057070554 wait complete. 2024-03-25T18:12:20.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:20.653+0000 7f7936d15640 1 librados: shutdown 2024-03-25T18:12:20.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:23.526 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:20.345 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.533 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:20.352 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.572 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:20.391 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.583 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:20.402 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.584 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:20.368 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.590 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:20.374 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.593 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:20.412 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.609 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:20.393 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:23.622 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:20.441 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:27.473 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.583+0000 7f5e08063640 10 monclient: get_monmap_and_config 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.583+0000 7f5e08063640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.583+0000 7f5e08063640 10 monclient: monmap: 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:27.585047+0000 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:27.585047+0000 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.585 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:27.586 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:27.586 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:27.586 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.584+0000 7f5e08063640 1 Processor -- start 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.584+0000 7f5e08063640 1 -- start start 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.584+0000 7f5e08063640 10 monclient: init 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-b con 0x7f5e000b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-a con 0x7f5e000b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e000af940 con 0x7f5e000b1730 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e000b0a70 con 0x7f5e0014be90 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e000b0d50 con 0x7f5e000b4d60 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.585+0000 7f5e08063640 10 monclient(hunting): authenticate will time out at 7391.884561s 2024-03-25T18:12:27.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.589+0000 7f5e05dff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5e000b4d60 0x7f5e0013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35068/0 (socket says 172.21.15.84:35068) 2024-03-25T18:12:27.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.590+0000 7f5e05dff640 1 -- 172.21.15.84:0/1933995751 learned_addr learned my addr 172.21.15.84:0/1933995751 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262253520 0 0) 0x7f5e000af940 con 0x7f5e000b1730 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:12:27.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.595+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec003520 con 0x7f5e000b1730 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529546696 0 0) 0x7f5e000b0d50 con 0x7f5e000b4d60 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:12:27.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.596+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e000af940 con 0x7f5e000b4d60 2024-03-25T18:12:27.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785974490 0 0) 0x7f5e000af940 con 0x7f5e000b4d60 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 >> v1:172.21.15.157:6789/0 conn(0x7f5e0014be90 legacy=0x7f5e0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 >> v1:172.21.15.84:6789/0 conn(0x7f5e000b1730 legacy=0x7f5e000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 monclient: found mon.noname-b 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e000b1030 con 0x7f5e000b4d60 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5dfc001670 con 0x7f5e000b4d60 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e04dfd640 10 monclient: dump: 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e08063640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:12:27.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.604+0000 7f5e08063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:27.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dfc0017d0 con 0x7f5e000b4d60 2024-03-25T18:12:27.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:27.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 1 -- 172.21.15.84:0/1933995751 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5dfc001ba0 con 0x7f5e000b4d60 2024-03-25T18:12:27.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e04dfd640 10 monclient: dump: 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e08063640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:27.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.609+0000 7f5e08063640 4 set_mon_vals no callback set 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 10 monclient: shutdown 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 1 -- 172.21.15.84:0/1933995751 >> v1:172.21.15.84:6790/0 conn(0x7f5e000b4d60 legacy=0x7f5e0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 1 -- 172.21.15.84:0/1933995751 shutdown_connections 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 1 -- 172.21.15.84:0/1933995751 >> 172.21.15.84:0/1933995751 conn(0x7f5e000617a0 msgr2=0x7f5e00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 1 -- 172.21.15.84:0/1933995751 shutdown_connections 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.610+0000 7f5e08063640 1 -- 172.21.15.84:0/1933995751 wait complete. 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 10 monclient: monmap: 2024-03-25T18:12:27.613 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 librados: starting msgr at 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 librados: starting objecter 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 Processor -- start 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 -- start start 2024-03-25T18:12:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 librados: setting wanted keys 2024-03-25T18:12:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 1 librados: calling monclient init 2024-03-25T18:12:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.611+0000 7f5e08063640 10 monclient: init 2024-03-25T18:12:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-b con 0x7f5e000b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-a con 0x7f5e000b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e001db0d0 con 0x7f5e000b4d60 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e001dbd00 con 0x7f5e000b1730 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e001dbe90 con 0x7f5e0014be90 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.612+0000 7f5e08063640 10 monclient(hunting): authenticate will time out at 7391.911552s 2024-03-25T18:12:27.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.619+0000 7f5e055fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5e000b4d60 0x7f5e0013b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:57092/0 (socket says 172.21.15.84:57092) 2024-03-25T18:12:27.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.619+0000 7f5e055fe640 1 -- 172.21.15.84:0/748937281 learned_addr learned my addr 172.21.15.84:0/748937281 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:27.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.622+0000 7f5df67fc640 10 client.?.objecter ms_handle_connect 0x7f5e000b4d60 2024-03-25T18:12:27.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.622+0000 7f5df67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.625+0000 7f5df67fc640 10 client.?.objecter ms_handle_connect 0x7f5e0014be90 2024-03-25T18:12:27.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.625+0000 7f5df67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.626+0000 7f5df67fc640 10 client.?.objecter ms_handle_connect 0x7f5e000b1730 2024-03-25T18:12:27.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.626+0000 7f5df67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332460549 0 0) 0x7f5e001db0d0 con 0x7f5e000b4d60 2024-03-25T18:12:27.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:12:27.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.633+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de4004eb0 con 0x7f5e000b4d60 2024-03-25T18:12:27.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365918032 0 0) 0x7f5e000b0a70 con 0x7f5e0014be90 2024-03-25T18:12:27.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:12:27.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.636+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e001db0d0 con 0x7f5e0014be90 2024-03-25T18:12:27.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647566146 0 0) 0x7f5de4004eb0 con 0x7f5e000b4d60 2024-03-25T18:12:27.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e000b0a70 con 0x7f5e000b4d60 2024-03-25T18:12:27.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5df0001990 con 0x7f5e000b4d60 2024-03-25T18:12:27.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): dump: 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232998725 0 0) 0x7f5e001dbd00 con 0x7f5e000b1730 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:12:27.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.641+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de4004eb0 con 0x7f5e000b1730 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198151104 0 0) 0x7f5e001db0d0 con 0x7f5e0014be90 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e001dbd00 con 0x7f5e0014be90 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5df80019b0 con 0x7f5e0014be90 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.647+0000 7f5df67fc640 10 monclient(hunting): dump: 2024-03-25T18:12:27.649 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:27.650 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 950152308 0 0) 0x7f5e000b0a70 con 0x7f5e000b4d60 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 >> v1:172.21.15.84:6790/0 conn(0x7f5e0014be90 legacy=0x7f5e001e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 >> v1:172.21.15.157:6789/0 conn(0x7f5e000b1730 legacy=0x7f5e001db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 1 monclient: found mon.a 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0013be30 con 0x7f5e000b4d60 2024-03-25T18:12:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5df67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 10 monclient: _renew_subs 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0013c080 con 0x7f5e000b4d60 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 10 monclient: _renew_subs 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e001399b0 con 0x7f5e000b4d60 2024-03-25T18:12:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.654+0000 7f5e08063640 1 librados: init done 2024-03-25T18:12:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.655+0000 7f5e08063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.655+0000 7f5e08063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.655+0000 7f5e08063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.655+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e001401f0 con 0x7f5e000b4d60 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5df00035f0 con 0x7f5e000b4d60 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5df00039c0 con 0x7f5e000b4d60 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df67fc640 10 monclient: dump: 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.949 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.947+0000 7f5df7fff640 4 set_mon_vals no callback set 2024-03-25T18:12:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.948+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f5df0003be0 con 0x7f5e000b4d60 2024-03-25T18:12:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.949+0000 7f5e05dff640 10 monclient: get_auth_request con 0x7f5de406dfb0 auth_method 0 2024-03-25T18:12:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.949+0000 7f5df67fc640 10 client.4557.objecter ms_handle_connect 0x7f5de406dfb0 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f5df0054750 con 0x7f5e000b4d60 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 10 client.4557.objecter ms_dispatch 0x7f5e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5df0050260 con 0x7f5e000b4d60 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:27.954+0000 7f5df67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:28.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.197+0000 7f5e08063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.197+0000 7f5e08063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.197+0000 7f5e08063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:12:28.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.197+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e000627b0 con 0x7f5e000b4d60 2024-03-25T18:12:28.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.238+0000 7f5df67fc640 1 -- 172.21.15.84:0/748937281 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f5df0052190 con 0x7f5e000b4d60 2024-03-25T18:12:28.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.238+0000 7f5df67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.238+0000 7f5df67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:28.240 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:28.240 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.240+0000 7f5e08063640 10 librados: watch_flush enter 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.240+0000 7f5e08063640 10 librados: watch_flush exit 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.240+0000 7f5e08063640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.240+0000 7f5e08063640 10 client.4557.objecter successfully canceled tick 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.240+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 >> v1:172.21.15.157:6800/1801259026 conn(0x7f5de406dfb0 legacy=0x7f5de4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 10 monclient: shutdown 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 >> v1:172.21.15.84:6789/0 conn(0x7f5e000b4d60 legacy=0x7f5e0013b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 shutdown_connections 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 >> 172.21.15.84:0/748937281 conn(0x7f5e000617a0 msgr2=0x7f5e0013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 shutdown_connections 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 -- 172.21.15.84:0/748937281 wait complete. 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:28.241+0000 7f5e08063640 1 librados: shutdown 2024-03-25T18:12:28.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:35.061 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:35.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.175+0000 7ff122f63640 10 monclient: get_monmap_and_config 2024-03-25T18:12:35.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.175+0000 7ff122f63640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.175+0000 7ff122f63640 10 monclient: monmap: 2024-03-25T18:12:35.176 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:35.176 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:35.176782+0000 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:35.176782+0000 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:35.177 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.176+0000 7ff122f63640 1 Processor -- start 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.176+0000 7ff122f63640 1 -- start start 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.176+0000 7ff122f63640 10 monclient: init 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-a con 0x7ff11c0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-b con 0x7ff11c0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-c con 0x7ff11c14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0af940 con 0x7ff11c0b4d60 2024-03-25T18:12:35.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0b0a70 con 0x7ff11c14be90 2024-03-25T18:12:35.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0b0d50 con 0x7ff11c0b1730 2024-03-25T18:12:35.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.177+0000 7ff122f63640 10 monclient(hunting): authenticate will time out at 7399.476011s 2024-03-25T18:12:35.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.178+0000 7ff121500640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff11c14be90 0x7ff11c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:54358/0 (socket says 172.21.15.84:54358) 2024-03-25T18:12:35.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.178+0000 7ff121500640 1 -- 172.21.15.84:0/317329002 learned_addr learned my addr 172.21.15.84:0/317329002 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113601931 0 0) 0x7ff11c0b0a70 con 0x7ff11c14be90 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:12:35.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.181+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff10c003520 con 0x7ff11c14be90 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756272465 0 0) 0x7ff10c003520 con 0x7ff11c14be90 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 >> v1:172.21.15.84:6790/0 conn(0x7ff11c0b1730 legacy=0x7ff11c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 >> v1:172.21.15.84:6789/0 conn(0x7ff11c0b4d60 legacy=0x7ff11c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0b1030 con 0x7ff11c14be90 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff114001a10 con 0x7ff11c14be90 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff11b7fe640 10 monclient: dump: 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff122f63640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:12:35.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.199+0000 7ff122f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff114001b70 con 0x7ff11c14be90 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 1 -- 172.21.15.84:0/317329002 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff114001f80 con 0x7ff11c14be90 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff11b7fe640 10 monclient: dump: 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.210 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff122f63640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:35.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.208+0000 7ff122f63640 4 set_mon_vals no callback set 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 10 monclient: shutdown 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 1 -- 172.21.15.84:0/317329002 >> v1:172.21.15.157:6789/0 conn(0x7ff11c14be90 legacy=0x7ff11c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 1 -- 172.21.15.84:0/317329002 shutdown_connections 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 1 -- 172.21.15.84:0/317329002 >> 172.21.15.84:0/317329002 conn(0x7ff11c0617a0 msgr2=0x7ff11c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 1 -- 172.21.15.84:0/317329002 shutdown_connections 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.209+0000 7ff122f63640 1 -- 172.21.15.84:0/317329002 wait complete. 2024-03-25T18:12:35.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.210+0000 7ff122f63640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.210+0000 7ff122f63640 10 monclient: monmap: 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 librados: starting msgr at 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 librados: starting objecter 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 Processor -- start 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 -- start start 2024-03-25T18:12:35.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 librados: setting wanted keys 2024-03-25T18:12:35.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 1 librados: calling monclient init 2024-03-25T18:12:35.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.211+0000 7ff122f63640 10 monclient: init 2024-03-25T18:12:35.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-b con 0x7ff11c0b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-c con 0x7ff11c0b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient(hunting): picked mon.noname-a con 0x7ff11c14be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c135ee0 con 0x7ff11c14be90 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c135b00 con 0x7ff11c0b1730 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c135c90 con 0x7ff11c0b4d60 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.212+0000 7ff122f63640 10 monclient(hunting): authenticate will time out at 7399.511089s 2024-03-25T18:12:35.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.221+0000 7ff121500640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff11c14be90 0x7ff11c1d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:50650/0 (socket says 172.21.15.84:50650) 2024-03-25T18:12:35.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.221+0000 7ff121500640 1 -- 172.21.15.84:0/1092642030 learned_addr learned my addr 172.21.15.84:0/1092642030 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:35.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.222+0000 7ff1197fa640 10 client.?.objecter ms_handle_connect 0x7ff11c14be90 2024-03-25T18:12:35.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.222+0000 7ff1197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.223+0000 7ff1197fa640 10 client.?.objecter ms_handle_connect 0x7ff11c0b4d60 2024-03-25T18:12:35.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.223+0000 7ff1197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.224+0000 7ff1197fa640 10 client.?.objecter ms_handle_connect 0x7ff11c0b1730 2024-03-25T18:12:35.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.224+0000 7ff1197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859860694 0 0) 0x7ff11c135ee0 con 0x7ff11c14be90 2024-03-25T18:12:35.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:12:35.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.228+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff100004eb0 con 0x7ff11c14be90 2024-03-25T18:12:35.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412179491 0 0) 0x7ff11c135c90 con 0x7ff11c0b4d60 2024-03-25T18:12:35.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:12:35.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.234+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c135ee0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032290634 0 0) 0x7ff11c135b00 con 0x7ff11c0b1730 2024-03-25T18:12:35.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:12:35.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.240+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c135c90 con 0x7ff11c0b1730 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1786382686 0 0) 0x7ff11c135ee0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff11c135b00 con 0x7ff11c0b4d60 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff1080018b0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 10 monclient(hunting): dump: 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:35.244 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:35.245 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:35.245 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:35.245 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.242+0000 7ff1197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 649745162 0 0) 0x7ff11c135b00 con 0x7ff11c0b4d60 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 >> v1:172.21.15.157:6789/0 conn(0x7ff11c0b1730 legacy=0x7ff11c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 >> v1:172.21.15.84:6789/0 conn(0x7ff11c14be90 legacy=0x7ff11c1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 1 monclient: found mon.c 2024-03-25T18:12:35.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c136960 con 0x7ff11c0b4d60 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff1197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 10 monclient: _renew_subs 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c130470 con 0x7ff11c0b4d60 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 10 monclient: _renew_subs 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c13e870 con 0x7ff11c0b4d60 2024-03-25T18:12:35.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.252+0000 7ff122f63640 1 librados: init done 2024-03-25T18:12:35.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.253+0000 7ff122f63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.253+0000 7ff122f63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.253+0000 7ff122f63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:35.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.253+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c130810 con 0x7ff11c0b4d60 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff108003610 con 0x7ff11c0b4d60 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff1080038c0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff1197fa640 10 monclient: dump: 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:35.568 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.566+0000 7ff11affd640 4 set_mon_vals no callback set 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.567+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7ff108003ae0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.567+0000 7ff120cff640 10 monclient: get_auth_request con 0x7ff10006de20 auth_method 0 2024-03-25T18:12:35.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.568+0000 7ff1197fa640 10 client.4556.objecter ms_handle_connect 0x7ff10006de20 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7ff1080533a0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 10 client.4556.objecter ms_dispatch 0x7ff11c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff10804eeb0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.574+0000 7ff1197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:35.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.824+0000 7ff122f63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:35.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.824+0000 7ff122f63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:35.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.824+0000 7ff122f63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:12:35.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.824+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff11c136af0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.850+0000 7ff1197fa640 1 -- 172.21.15.84:0/1092642030 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7ff108050de0 con 0x7ff11c0b4d60 2024-03-25T18:12:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.850+0000 7ff1197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.850+0000 7ff1197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:35.852 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:35.852 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 10 librados: watch_flush enter 2024-03-25T18:12:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 10 librados: watch_flush exit 2024-03-25T18:12:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 10 client.4556.objecter successfully canceled tick 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 >> v1:172.21.15.157:6800/1801259026 conn(0x7ff10006de20 legacy=0x7ff1000702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 10 monclient: shutdown 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.852+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 >> v1:172.21.15.84:6790/0 conn(0x7ff11c0b4d60 legacy=0x7ff11c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.853+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 shutdown_connections 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.853+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 >> 172.21.15.84:0/1092642030 conn(0x7ff11c0617a0 msgr2=0x7ff11c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.853+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 shutdown_connections 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.853+0000 7ff122f63640 1 -- 172.21.15.84:0/1092642030 wait complete. 2024-03-25T18:12:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:35.853+0000 7ff122f63640 1 librados: shutdown 2024-03-25T18:12:35.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:42.671 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.785+0000 7f8b908d9640 10 monclient: get_monmap_and_config 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.785+0000 7f8b908d9640 10 monclient: build_initial_monmap 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.785+0000 7f8b908d9640 10 monclient: monmap: 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:42.787130+0000 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:42.787130+0000 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:42.787 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:42.788 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:42.788 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:42.788 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.786+0000 7f8b908d9640 1 Processor -- start 2024-03-25T18:12:42.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.786+0000 7f8b908d9640 1 -- start start 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.786+0000 7f8b908d9640 10 monclient: init 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c06ab80 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c06af60 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8c0b00b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c0b0500 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c13ffa0 con 0x7f8b8c06af60 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.787+0000 7f8b908d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c0af520 con 0x7f8b8c06ab80 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.788+0000 7f8b908d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:42.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.788+0000 7f8b908d9640 10 monclient(hunting): authenticate will time out at 7407.086934s 2024-03-25T18:12:42.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.793+0000 7f8b8ad9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f8b8c0b00b0 0x7f8b8c14dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:40056/0 (socket says 172.21.15.84:40056) 2024-03-25T18:12:42.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.793+0000 7f8b8ad9d640 1 -- 172.21.15.84:0/1123156598 learned_addr learned my addr 172.21.15.84:0/1123156598 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561492176 0 0) 0x7f8b8c0b0500 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:12:42.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.805+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b74003520 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140069198 0 0) 0x7f8b8c13ffa0 con 0x7f8b8c06af60 2024-03-25T18:12:42.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:12:42.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.815+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c0b0500 con 0x7f8b8c06af60 2024-03-25T18:12:42.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070342059 0 0) 0x7f8b8c0af520 con 0x7f8b8c06ab80 2024-03-25T18:12:42.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:12:42.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.821+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c13ffa0 con 0x7f8b8c06ab80 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650962973 0 0) 0x7f8b8c0b0500 con 0x7f8b8c06af60 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 >> v1:172.21.15.84:6790/0 conn(0x7f8b8c06ab80 legacy=0x7f8b8c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 >> v1:172.21.15.84:6789/0 conn(0x7f8b8c0b00b0 legacy=0x7f8b8c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 monclient: found mon.noname-c 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:42.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c072890 con 0x7f8b8c06af60 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: _finish_auth 0 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8b80001670 con 0x7f8b8c06af60 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b8959a640 10 monclient: dump: 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:42.829 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:42.830 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:42.830 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:42.830 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b908d9640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:12:42.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.826+0000 7f8b908d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b800017d0 con 0x7f8b8c06af60 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 1 -- 172.21.15.84:0/1123156598 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8b80001be0 con 0x7f8b8c06af60 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b908d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b908d9640 4 set_mon_vals no callback set 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.834+0000 7f8b8959a640 10 monclient: dump: 2024-03-25T18:12:42.836 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:42.837 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 10 monclient: shutdown 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 1 -- 172.21.15.84:0/1123156598 >> v1:172.21.15.157:6789/0 conn(0x7f8b8c06af60 legacy=0x7f8b8c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 1 -- 172.21.15.84:0/1123156598 shutdown_connections 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 1 -- 172.21.15.84:0/1123156598 >> 172.21.15.84:0/1123156598 conn(0x7f8b8c0b4a80 msgr2=0x7f8b8c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 1 -- 172.21.15.84:0/1123156598 shutdown_connections 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.835+0000 7f8b908d9640 1 -- 172.21.15.84:0/1123156598 wait complete. 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.836+0000 7f8b908d9640 10 monclient: build_initial_monmap 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.836+0000 7f8b908d9640 10 monclient: monmap: 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:42.839 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 librados: starting msgr at 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 librados: starting objecter 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 Processor -- start 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 -- start start 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 librados: setting wanted keys 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 1 librados: calling monclient init 2024-03-25T18:12:42.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.837+0000 7f8b908d9640 10 monclient: init 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c06ab80 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8c0b00b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c138600 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c1d8420 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c1d8a00 con 0x7f8b8c06ab80 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c1d8b90 con 0x7f8b8c138600 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:42.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.838+0000 7f8b908d9640 10 monclient(hunting): authenticate will time out at 7407.137276s 2024-03-25T18:12:42.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.845+0000 7f8b8a59c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f8b8c06ab80 0x7f8b8c1460e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:57166/0 (socket says 172.21.15.84:57166) 2024-03-25T18:12:42.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.845+0000 7f8b8a59c640 1 -- 172.21.15.84:0/2758224183 learned_addr learned my addr 172.21.15.84:0/2758224183 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:42.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.845+0000 7f8b7affd640 10 client.?.objecter ms_handle_connect 0x7f8b8c06ab80 2024-03-25T18:12:42.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.845+0000 7f8b7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:42.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.854+0000 7f8b7affd640 10 client.?.objecter ms_handle_connect 0x7f8b8c138600 2024-03-25T18:12:42.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.854+0000 7f8b7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:42.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.855+0000 7f8b7affd640 10 client.?.objecter ms_handle_connect 0x7f8b8c0b00b0 2024-03-25T18:12:42.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.855+0000 7f8b7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743127465 0 0) 0x7f8b8c1d8420 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:12:42.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c004eb0 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844093223 0 0) 0x7f8b8c1d8a00 con 0x7f8b8c06ab80 2024-03-25T18:12:42.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:12:42.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.858+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c1d8420 con 0x7f8b8c06ab80 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473010390 0 0) 0x7f8b8c1d8b90 con 0x7f8b8c138600 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:12:42.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.864+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c1d8a00 con 0x7f8b8c138600 2024-03-25T18:12:42.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230204952 0 0) 0x7f8b8c1d8420 con 0x7f8b8c06ab80 2024-03-25T18:12:42.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b8c1d8b90 con 0x7f8b8c06ab80 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8b7c0019f0 con 0x7f8b8c06ab80 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 10 monclient(hunting): dump: 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:42.875 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.873+0000 7f8b7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:42.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2661068017 0 0) 0x7f8b6c004eb0 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b8c1d8420 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8b80001ba0 con 0x7f8b8c0b00b0 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.879+0000 7f8b7affd640 10 monclient(hunting): dump: 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:42.881 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:42.882 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:42.882 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:42.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1909623263 0 0) 0x7f8b8c1d8b90 con 0x7f8b8c06ab80 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 >> v1:172.21.15.84:6790/0 conn(0x7f8b8c138600 legacy=0x7f8b8c144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 >> v1:172.21.15.84:6789/0 conn(0x7f8b8c0b00b0 legacy=0x7f8b8c1d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 1 monclient: found mon.b 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c144aa0 con 0x7f8b8c06ab80 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 10 monclient: _finish_auth 0 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:42.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b908d9640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.882+0000 7f8b908d9640 10 monclient: _renew_subs 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c143660 con 0x7f8b8c06ab80 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 10 monclient: _renew_subs 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c143b90 con 0x7f8b8c06ab80 2024-03-25T18:12:42.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.883+0000 7f8b908d9640 1 librados: init done 2024-03-25T18:12:42.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.884+0000 7f8b908d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:42.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.884+0000 7f8b908d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:42.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.884+0000 7f8b908d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:42.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:42.884+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c144c90 con 0x7f8b8c06ab80 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b7c003610 con 0x7f8b8c06ab80 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f8b7c003920 con 0x7f8b8c06ab80 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b7affd640 10 monclient: dump: 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.181 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.179+0000 7f8b88d99640 4 set_mon_vals no callback set 2024-03-25T18:12:43.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.180+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f8b7c003b40 con 0x7f8b8c06ab80 2024-03-25T18:12:43.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.181+0000 7f8b89d9b640 10 monclient: get_auth_request con 0x7f8b6c06df70 auth_method 0 2024-03-25T18:12:43.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.181+0000 7f8b7affd640 10 client.4570.objecter ms_handle_connect 0x7f8b6c06df70 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.185+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f8b7c0547e0 con 0x7f8b8c06ab80 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.185+0000 7f8b7affd640 10 client.4570.objecter ms_dispatch 0x7f8b8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.185+0000 7f8b7affd640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.185+0000 7f8b7affd640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.185+0000 7f8b7affd640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.186+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b7c0502f0 con 0x7f8b8c06ab80 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.186+0000 7f8b7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.186+0000 7f8b7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:43.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.433+0000 7f8b908d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.433+0000 7f8b908d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.433+0000 7f8b908d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:43.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.433+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b8c0627b0 con 0x7f8b8c06ab80 2024-03-25T18:12:43.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.490+0000 7f8b7affd640 1 -- 172.21.15.84:0/2758224183 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f8b7c052220 con 0x7f8b8c06ab80 2024-03-25T18:12:43.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.490+0000 7f8b7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.490+0000 7f8b7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:43.492 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:43.492 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 10 librados: watch_flush enter 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 10 librados: watch_flush exit 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 >> v1:172.21.15.157:6800/1801259026 conn(0x7f8b6c06df70 legacy=0x7f8b6c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 10 monclient: shutdown 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 >> v1:172.21.15.157:6789/0 conn(0x7f8b8c06ab80 legacy=0x7f8b8c1460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 shutdown_connections 2024-03-25T18:12:43.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 >> 172.21.15.84:0/2758224183 conn(0x7f8b8c0b4a80 msgr2=0x7f8b8c13e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 shutdown_connections 2024-03-25T18:12:43.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 -- 172.21.15.84:0/2758224183 wait complete. 2024-03-25T18:12:43.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:43.493+0000 7f8b908d9640 1 librados: shutdown 2024-03-25T18:12:43.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:43.580 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:40.399 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.632 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:40.451 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.661 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:07:40.480 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.697 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:40.481 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.699 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:40.518 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.704 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:40.487 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.710 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:40.529 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.736 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:40.555 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:43.737 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:07:40.520 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.310 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.424+0000 7f388d9db640 10 monclient: get_monmap_and_config 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.424+0000 7f388d9db640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.424+0000 7f388d9db640 10 monclient: monmap: 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:50.425223+0000 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:50.425223+0000 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.425 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:50.426 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:50.426 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:50.426 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:50.426 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.425+0000 7f388d9db640 1 Processor -- start 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.425+0000 7f388d9db640 1 -- start start 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.425+0000 7f388d9db640 10 monclient: init 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-a con 0x7f38880a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-c con 0x7f388813f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-b con 0x7f38881430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888142520 con 0x7f38880a98f0 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880b28a0 con 0x7f388813f710 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880b2a30 con 0x7f38881430c0 2024-03-25T18:12:50.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.426+0000 7f388d9db640 10 monclient(hunting): authenticate will time out at 7414.724223s 2024-03-25T18:12:50.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.433+0000 7f38877fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f38881430c0 0x7f388814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43036/0 (socket says 172.21.15.84:43036) 2024-03-25T18:12:50.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.433+0000 7f38877fe640 1 -- 172.21.15.84:0/734645918 learned_addr learned my addr 172.21.15.84:0/734645918 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164256870 0 0) 0x7f38880b2a30 con 0x7f38881430c0 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:12:50.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.438+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874003520 con 0x7f38881430c0 2024-03-25T18:12:50.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19104929 0 0) 0x7f3874003520 con 0x7f38881430c0 2024-03-25T18:12:50.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 >> v1:172.21.15.157:6789/0 conn(0x7f388813f710 legacy=0x7f38881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:12:50.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 >> v1:172.21.15.84:6789/0 conn(0x7f38880a98f0 legacy=0x7f38880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 1 monclient: found mon.noname-b 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38880b44c0 con 0x7f38881430c0 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f38867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f388d9db640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:12:50.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.442+0000 7f388d9db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:50.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.445+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3880001a10 con 0x7f38881430c0 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.445+0000 7f38867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.445+0000 7f38867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.445+0000 7f38867fc640 10 monclient: dump: 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:50.446 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:50.447 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.445+0000 7f388d9db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:50.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3880001ce0 con 0x7f38881430c0 2024-03-25T18:12:50.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 1 -- 172.21.15.84:0/734645918 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3880002070 con 0x7f38881430c0 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f38867fc640 10 monclient: dump: 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f388d9db640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:50.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.455+0000 7f388d9db640 4 set_mon_vals no callback set 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 10 monclient: shutdown 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 1 -- 172.21.15.84:0/734645918 >> v1:172.21.15.84:6790/0 conn(0x7f38881430c0 legacy=0x7f388814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 1 -- 172.21.15.84:0/734645918 shutdown_connections 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 1 -- 172.21.15.84:0/734645918 >> 172.21.15.84:0/734645918 conn(0x7f38880b7ea0 msgr2=0x7f38880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 1 -- 172.21.15.84:0/734645918 shutdown_connections 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.456+0000 7f388d9db640 1 -- 172.21.15.84:0/734645918 wait complete. 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 10 monclient: monmap: 2024-03-25T18:12:50.458 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 librados: starting msgr at 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 librados: starting objecter 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 Processor -- start 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 -- start start 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 librados: setting wanted keys 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 1 librados: calling monclient init 2024-03-25T18:12:50.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.457+0000 7f388d9db640 10 monclient: init 2024-03-25T18:12:50.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-b con 0x7f38880a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-a con 0x7f388813f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient(hunting): picked mon.noname-c con 0x7f38881430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888137dd0 con 0x7f388813f710 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38881349e0 con 0x7f38880a98f0 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38881384a0 con 0x7f38881430c0 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.458+0000 7f388d9db640 10 monclient(hunting): authenticate will time out at 7414.756640s 2024-03-25T18:12:50.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.462+0000 7f3886ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f38880a98f0 0x7f38881da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51280/0 (socket says 172.21.15.84:51280) 2024-03-25T18:12:50.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.462+0000 7f3886ffd640 1 -- 172.21.15.84:0/1748160442 learned_addr learned my addr 172.21.15.84:0/1748160442 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:50.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.466+0000 7f3884ff9640 10 client.?.objecter ms_handle_connect 0x7f38880a98f0 2024-03-25T18:12:50.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.466+0000 7f3884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.472+0000 7f3884ff9640 10 client.?.objecter ms_handle_connect 0x7f38881430c0 2024-03-25T18:12:50.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.472+0000 7f3884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.475+0000 7f3884ff9640 10 client.?.objecter ms_handle_connect 0x7f388813f710 2024-03-25T18:12:50.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.475+0000 7f3884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009145274 0 0) 0x7f38881349e0 con 0x7f38880a98f0 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:12:50.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.478+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868004eb0 con 0x7f38880a98f0 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530877319 0 0) 0x7f38881384a0 con 0x7f38881430c0 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:12:50.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.490+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38881349e0 con 0x7f38881430c0 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739018928 0 0) 0x7f3888137dd0 con 0x7f388813f710 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38881384a0 con 0x7f388813f710 2024-03-25T18:12:50.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931292938 0 0) 0x7f3868004eb0 con 0x7f38880a98f0 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3888137dd0 con 0x7f38880a98f0 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f38780019d0 con 0x7f38880a98f0 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.491+0000 7f3884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:50.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4188508730 0 0) 0x7f3888137dd0 con 0x7f38880a98f0 2024-03-25T18:12:50.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 >> v1:172.21.15.84:6790/0 conn(0x7f38881430c0 legacy=0x7f38881342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 >> v1:172.21.15.84:6789/0 conn(0x7f388813f710 legacy=0x7f3888133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 1 monclient: found mon.b 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888139980 con 0x7f38880a98f0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f3884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f388d9db640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f388d9db640 10 monclient: _renew_subs 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f388d9db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.498+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38881389c0 con 0x7f38880a98f0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.499+0000 7f388d9db640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.499+0000 7f388d9db640 10 monclient: _renew_subs 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.499+0000 7f388d9db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.499+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3888137fc0 con 0x7f38880a98f0 2024-03-25T18:12:50.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.499+0000 7f388d9db640 1 librados: init done 2024-03-25T18:12:50.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.500+0000 7f388d9db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.500+0000 7f388d9db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.500+0000 7f388d9db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:50.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.500+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38881382c0 con 0x7f38880a98f0 2024-03-25T18:12:50.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3878001c90 con 0x7f38880a98f0 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3878003a60 con 0x7f38880a98f0 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3884ff9640 10 monclient: dump: 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:50.806 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.805+0000 7f3885ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.806+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f3878003c80 con 0x7f38880a98f0 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.806+0000 7f387e7fc640 10 monclient: get_auth_request con 0x7f386806de00 auth_method 0 2024-03-25T18:12:50.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.807+0000 7f3884ff9640 10 client.4573.objecter ms_handle_connect 0x7f386806de00 2024-03-25T18:12:50.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f38780547b0 con 0x7f38880a98f0 2024-03-25T18:12:50.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 10 client.4573.objecter ms_dispatch 0x7f3888000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38780502c0 con 0x7f38880a98f0 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:50.811+0000 7f3884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:51.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.060+0000 7f388d9db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.060+0000 7f388d9db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.060+0000 7f388d9db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:51.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.060+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38880627b0 con 0x7f38880a98f0 2024-03-25T18:12:51.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.163+0000 7f3884ff9640 1 -- 172.21.15.84:0/1748160442 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f38780521f0 con 0x7f38880a98f0 2024-03-25T18:12:51.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.163+0000 7f3884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.163+0000 7f3884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:51.163 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:51.164 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 10 librados: watch_flush enter 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 10 librados: watch_flush exit 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 10 client.4573.objecter successfully canceled tick 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 >> v1:172.21.15.157:6800/1801259026 conn(0x7f386806de00 legacy=0x7f38680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 10 monclient: shutdown 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 >> v1:172.21.15.157:6789/0 conn(0x7f38880a98f0 legacy=0x7f38881da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 shutdown_connections 2024-03-25T18:12:51.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 >> 172.21.15.84:0/1748160442 conn(0x7f38880b7ea0 msgr2=0x7f3888074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:51.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 shutdown_connections 2024-03-25T18:12:51.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 -- 172.21.15.84:0/1748160442 wait complete. 2024-03-25T18:12:51.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:51.165+0000 7f388d9db640 1 librados: shutdown 2024-03-25T18:12:51.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:57.984 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.094+0000 7fa36313b640 10 monclient: get_monmap_and_config 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.094+0000 7fa36313b640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.094+0000 7fa36313b640 10 monclient: monmap: 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:12:58.095126+0000 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:12:58.095126+0000 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.095 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:58.096 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:12:58.096 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:12:58.096 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.095+0000 7fa36313b640 1 Processor -- start 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.095+0000 7fa36313b640 1 -- start start 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.095+0000 7fa36313b640 10 monclient: init 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-a con 0x7fa35c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-c con 0x7fa35c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-b con 0x7fa35c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c142520 con 0x7fa35c0a98f0 2024-03-25T18:12:58.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0b28a0 con 0x7fa35c13f710 2024-03-25T18:12:58.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0b2a30 con 0x7fa35c1430c0 2024-03-25T18:12:58.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.096+0000 7fa36313b640 10 monclient(hunting): authenticate will time out at 7422.394291s 2024-03-25T18:12:58.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.110+0000 7fa35bfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa35c13f710 0x7fa35c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56344/0 (socket says 172.21.15.84:56344) 2024-03-25T18:12:58.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.110+0000 7fa35bfff640 1 -- 172.21.15.84:0/1148743594 learned_addr learned my addr 172.21.15.84:0/1148743594 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.127+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444429959 0 0) 0x7fa35c0b28a0 con 0x7fa35c13f710 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.127+0000 7fa35b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.127+0000 7fa35b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.127+0000 7fa35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.127+0000 7fa35b7fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:12:58.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.128+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c003520 con 0x7fa35c13f710 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873335331 0 0) 0x7fa35c0b2a30 con 0x7fa35c1430c0 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c0b28a0 con 0x7fa35c1430c0 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513340115 0 0) 0x7fa35c142520 con 0x7fa35c0a98f0 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:12:58.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.137+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c0b2a30 con 0x7fa35c0a98f0 2024-03-25T18:12:58.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988693540 0 0) 0x7fa35c0b2a30 con 0x7fa35c0a98f0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 >> v1:172.21.15.84:6790/0 conn(0x7fa35c1430c0 legacy=0x7fa35c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 >> v1:172.21.15.157:6789/0 conn(0x7fa35c13f710 legacy=0x7fa35c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0b44c0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa350001530 con 0x7fa35c0a98f0 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa35b7fe640 10 monclient: dump: 2024-03-25T18:12:58.152 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa36313b640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:12:58.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.151+0000 7fa36313b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa350001690 con 0x7fa35c0a98f0 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 1 -- 172.21.15.84:0/1148743594 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa350001a60 con 0x7fa35c0a98f0 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:58.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa35b7fe640 10 monclient: dump: 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa36313b640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:58.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.168+0000 7fa36313b640 4 set_mon_vals no callback set 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 10 monclient: shutdown 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 1 -- 172.21.15.84:0/1148743594 >> v1:172.21.15.84:6789/0 conn(0x7fa35c0a98f0 legacy=0x7fa35c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 1 -- 172.21.15.84:0/1148743594 shutdown_connections 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 1 -- 172.21.15.84:0/1148743594 >> 172.21.15.84:0/1148743594 conn(0x7fa35c0b7ea0 msgr2=0x7fa35c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 1 -- 172.21.15.84:0/1148743594 shutdown_connections 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 1 -- 172.21.15.84:0/1148743594 wait complete. 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.169+0000 7fa36313b640 10 monclient: monmap: 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:12:58.171 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 librados: starting msgr at 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 librados: starting objecter 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 Processor -- start 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 -- start start 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 librados: setting wanted keys 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 1 librados: calling monclient init 2024-03-25T18:12:58.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.170+0000 7fa36313b640 10 monclient: init 2024-03-25T18:12:58.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-b con 0x7fa35c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-a con 0x7fa35c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient(hunting): picked mon.noname-c con 0x7fa35c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c1daa20 con 0x7fa35c13f710 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c1371e0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c1374c0 con 0x7fa35c1430c0 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.171+0000 7fa36313b640 10 monclient(hunting): authenticate will time out at 7422.469332s 2024-03-25T18:12:58.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.178+0000 7fa3616d8640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa35c1430c0 0x7fa35c1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48086/0 (socket says 172.21.15.84:48086) 2024-03-25T18:12:58.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.178+0000 7fa3616d8640 1 -- 172.21.15.84:0/3572129766 learned_addr learned my addr 172.21.15.84:0/3572129766 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:12:58.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.179+0000 7fa3597fa640 10 client.?.objecter ms_handle_connect 0x7fa35c1430c0 2024-03-25T18:12:58.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.179+0000 7fa3597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.183+0000 7fa3597fa640 10 client.?.objecter ms_handle_connect 0x7fa35c0a98f0 2024-03-25T18:12:58.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.183+0000 7fa3597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.191+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654481544 0 0) 0x7fa35c1374c0 con 0x7fa35c1430c0 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.192+0000 7fa3597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.192+0000 7fa3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.192+0000 7fa3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.192+0000 7fa3597fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:12:58.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.192+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340004eb0 con 0x7fa35c1430c0 2024-03-25T18:12:58.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.193+0000 7fa3597fa640 10 client.?.objecter ms_handle_connect 0x7fa35c13f710 2024-03-25T18:12:58.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.193+0000 7fa3597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272199081 0 0) 0x7fa35c1371e0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:12:58.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.198+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c1374c0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021742119 0 0) 0x7fa35c1daa20 con 0x7fa35c13f710 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c1371e0 con 0x7fa35c13f710 2024-03-25T18:12:58.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.201+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007003539 0 0) 0x7fa340004eb0 con 0x7fa35c1430c0 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa35c1daa20 con 0x7fa35c1430c0 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa354001c90 con 0x7fa35c1430c0 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 10 monclient(hunting): dump: 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:58.203 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.202+0000 7fa3597fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853275387 0 0) 0x7fa35c1374c0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa340004eb0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa3500019a0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.206+0000 7fa3597fa640 10 monclient(hunting): dump: 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.207 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:58.208 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:58.208 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:58.208 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2067360540 0 0) 0x7fa340004eb0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 >> v1:172.21.15.84:6790/0 conn(0x7fa35c1430c0 legacy=0x7fa35c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 >> v1:172.21.15.84:6789/0 conn(0x7fa35c13f710 legacy=0x7fa35c1daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 1 monclient: found mon.b 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c13c040 con 0x7fa35c0a98f0 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa3597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 10 monclient: _renew_subs 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c1359f0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 10 monclient: _renew_subs 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c1d75d0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.215+0000 7fa36313b640 1 librados: init done 2024-03-25T18:12:58.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.216+0000 7fa36313b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.216+0000 7fa36313b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.216+0000 7fa36313b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:58.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.216+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c0627b0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa350001b00 con 0x7fa35c0a98f0 2024-03-25T18:12:58.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fa3500010f0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa3597fa640 10 monclient: dump: 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:12:58.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.501+0000 7fa35affd640 4 set_mon_vals no callback set 2024-03-25T18:12:58.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.502+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fa350019970 con 0x7fa35c0a98f0 2024-03-25T18:12:58.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.503+0000 7fa35bfff640 10 monclient: get_auth_request con 0x7fa34006df70 auth_method 0 2024-03-25T18:12:58.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.503+0000 7fa3597fa640 10 client.4579.objecter ms_handle_connect 0x7fa34006df70 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.507+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fa350053660 con 0x7fa35c0a98f0 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.507+0000 7fa3597fa640 10 client.4579.objecter ms_dispatch 0x7fa35c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.507+0000 7fa3597fa640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.507+0000 7fa3597fa640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.507+0000 7fa3597fa640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.508+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa35004efa0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.508+0000 7fa3597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.508+0000 7fa3597fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:58.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.726+0000 7fa36313b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:58.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.726+0000 7fa36313b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:58.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.726+0000 7fa36313b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:12:58.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.726+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa35c14f240 con 0x7fa35c0a98f0 2024-03-25T18:12:58.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.757+0000 7fa3597fa640 1 -- 172.21.15.84:0/3572129766 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fa3500510a0 con 0x7fa35c0a98f0 2024-03-25T18:12:58.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.757+0000 7fa3597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:58.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.757+0000 7fa3597fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:58.758 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:12:58.758 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:58.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 10 librados: watch_flush enter 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 10 librados: watch_flush exit 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 >> v1:172.21.15.157:6800/1801259026 conn(0x7fa34006df70 legacy=0x7fa340070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 10 monclient: shutdown 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.759+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 >> v1:172.21.15.157:6789/0 conn(0x7fa35c0a98f0 legacy=0x7fa35c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.760+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 shutdown_connections 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.760+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 >> 172.21.15.84:0/3572129766 conn(0x7fa35c0b7ea0 msgr2=0x7fa35c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.760+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 shutdown_connections 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.760+0000 7fa36313b640 1 -- 172.21.15.84:0/3572129766 wait complete. 2024-03-25T18:12:58.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:12:58.760+0000 7fa36313b640 1 librados: shutdown 2024-03-25T18:12:58.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:03.645 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:00.463 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.692 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:00.511 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.721 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:00.541 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.738 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:00.522 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.744 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:00.528 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.764 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:00.583 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.769 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:00.588 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.781 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:00.564 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.801 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:00.620 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:05.580 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.695+0000 7f1afd2d9640 10 monclient: get_monmap_and_config 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.695+0000 7f1afd2d9640 10 monclient: build_initial_monmap 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.695+0000 7f1afd2d9640 10 monclient: monmap: 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:05.696164+0000 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:05.696164+0000 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:05.696 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:05.697 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:05.697 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.696+0000 7f1afd2d9640 1 Processor -- start 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.696+0000 7f1afd2d9640 1 -- start start 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.696+0000 7f1afd2d9640 10 monclient: init 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1af80a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1af813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1af81430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af8142520 con 0x7f1af80a98f0 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80b28a0 con 0x7f1af81430c0 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80b2a30 con 0x7f1af813f710 2024-03-25T18:13:05.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:05.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.697+0000 7f1afd2d9640 10 monclient(hunting): authenticate will time out at 7429.995009s 2024-03-25T18:13:05.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.707+0000 7f1aee7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1af813f710 0x7f1af81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56972/0 (socket says 172.21.15.84:56972) 2024-03-25T18:13:05.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.707+0000 7f1aee7fc640 1 -- 172.21.15.84:0/1191836199 learned_addr learned my addr 172.21.15.84:0/1191836199 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:05.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.717+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732304668 0 0) 0x7f1af8142520 con 0x7f1af80a98f0 2024-03-25T18:13:05.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.718+0000 7f1af67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:05.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.718+0000 7f1af67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:05.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.718+0000 7f1af67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:05.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.718+0000 7f1af67fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:13:05.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.718+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae4003520 con 0x7f1af80a98f0 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776297914 0 0) 0x7f1af80b2a30 con 0x7f1af813f710 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:13:05.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.722+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8142520 con 0x7f1af813f710 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106229201 0 0) 0x7f1af8142520 con 0x7f1af813f710 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 >> v1:172.21.15.157:6789/0 conn(0x7f1af81430c0 legacy=0x7f1af814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 >> v1:172.21.15.84:6789/0 conn(0x7f1af80a98f0 legacy=0x7f1af80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 monclient: found mon.noname-b 2024-03-25T18:13:05.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af80b44c0 con 0x7f1af813f710 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ae0001670 con 0x7f1af813f710 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1afd2d9640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1afd2d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1af67fc640 10 monclient: dump: 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:05.802 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:05.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.800+0000 7f1afd2d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.817+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae00017d0 con 0x7f1af813f710 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.817+0000 7f1af67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.817+0000 7f1af67fc640 1 -- 172.21.15.84:0/1191836199 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ae0001ba0 con 0x7f1af813f710 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.817+0000 7f1af67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1af67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1af67fc640 10 monclient: dump: 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:05.818 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1afd2d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:05.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1afd2d9640 4 set_mon_vals no callback set 2024-03-25T18:13:05.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1afd2d9640 10 monclient: shutdown 2024-03-25T18:13:05.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1afd2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.818+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1191836199 >> v1:172.21.15.84:6790/0 conn(0x7f1af813f710 legacy=0x7f1af81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1191836199 shutdown_connections 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1191836199 >> 172.21.15.84:0/1191836199 conn(0x7f1af80b7ea0 msgr2=0x7f1af80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1191836199 shutdown_connections 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1191836199 wait complete. 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 10 monclient: build_initial_monmap 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.819+0000 7f1afd2d9640 10 monclient: monmap: 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:05.821 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 librados: starting msgr at 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 librados: starting objecter 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 Processor -- start 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 -- start start 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 librados: setting wanted keys 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 1 librados: calling monclient init 2024-03-25T18:13:05.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.820+0000 7f1afd2d9640 10 monclient: init 2024-03-25T18:13:05.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:05.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:05.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1af80a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:05.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1af813f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1af81430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80ab9e0 con 0x7f1af813f710 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af81d7830 con 0x7f1af81430c0 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af81d7500 con 0x7f1af80a98f0 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:05.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.821+0000 7f1afd2d9640 10 monclient(hunting): authenticate will time out at 7430.119479s 2024-03-25T18:13:05.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.834+0000 7f1af6ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1af80a98f0 0x7f1af8137b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57002/0 (socket says 172.21.15.84:57002) 2024-03-25T18:13:05.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.834+0000 7f1af6ffd640 1 -- 172.21.15.84:0/1849795149 learned_addr learned my addr 172.21.15.84:0/1849795149 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:05.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.834+0000 7f1aeffff640 10 client.?.objecter ms_handle_connect 0x7f1af80a98f0 2024-03-25T18:13:05.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.834+0000 7f1aeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:05.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223472959 0 0) 0x7f1af81d7500 con 0x7f1af80a98f0 2024-03-25T18:13:05.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:05.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:05.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:05.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:13:05.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.842+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8004eb0 con 0x7f1af80a98f0 2024-03-25T18:13:05.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573147868 0 0) 0x7f1ad8004eb0 con 0x7f1af80a98f0 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af81d7500 con 0x7f1af80a98f0 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ae8001a10 con 0x7f1af80a98f0 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 10 monclient(hunting): dump: 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:05.859 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:05.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.858+0000 7f1aeffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:05.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.865+0000 7f1aeffff640 10 client.?.objecter ms_handle_connect 0x7f1af81430c0 2024-03-25T18:13:05.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.865+0000 7f1aeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:05.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.874+0000 7f1aeffff640 10 client.?.objecter ms_handle_connect 0x7f1af813f710 2024-03-25T18:13:05.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.874+0000 7f1aeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288562573 0 0) 0x7f1af80ab9e0 con 0x7f1af813f710 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:13:05.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.886+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8004eb0 con 0x7f1af813f710 2024-03-25T18:13:05.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765821092 0 0) 0x7f1af81d7830 con 0x7f1af81430c0 2024-03-25T18:13:05.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:05.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:05.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:05.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:13:05.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.887+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af80ab9e0 con 0x7f1af81430c0 2024-03-25T18:13:05.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3156285419 0 0) 0x7f1af81d7500 con 0x7f1af80a98f0 2024-03-25T18:13:05.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 >> v1:172.21.15.157:6789/0 conn(0x7f1af81430c0 legacy=0x7f1af8134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 >> v1:172.21.15.84:6789/0 conn(0x7f1af813f710 legacy=0x7f1af81d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 1 monclient: found mon.c 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af8134790 con 0x7f1af80a98f0 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1aeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1afd2d9640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1afd2d9640 10 monclient: _renew_subs 2024-03-25T18:13:05.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1afd2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.897+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af81349e0 con 0x7f1af80a98f0 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 10 monclient: _renew_subs 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af8138b10 con 0x7f1af80a98f0 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 1 librados: init done 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.898+0000 7f1afd2d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.899+0000 7f1afd2d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:05.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.899+0000 7f1afd2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:05.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:05.899+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af8138dc0 con 0x7f1af80a98f0 2024-03-25T18:13:06.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae8001d10 con 0x7f1af80a98f0 2024-03-25T18:13:06.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:06.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1ae80038c0 con 0x7f1af80a98f0 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1aeffff640 10 monclient: dump: 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:06.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.195+0000 7f1af5ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:06.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.196+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1ae8003ae0 con 0x7f1af80a98f0 2024-03-25T18:13:06.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.196+0000 7f1aee7fc640 10 monclient: get_auth_request con 0x7f1ad806ddd0 auth_method 0 2024-03-25T18:13:06.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.197+0000 7f1aeffff640 10 client.4580.objecter ms_handle_connect 0x7f1ad806ddd0 2024-03-25T18:13:06.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1ae80547d0 con 0x7f1af80a98f0 2024-03-25T18:13:06.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 10 client.4580.objecter ms_dispatch 0x7f1af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ae80502e0 con 0x7f1af80a98f0 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.202+0000 7f1aeffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:06.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.450+0000 7f1afd2d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.450+0000 7f1afd2d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.450+0000 7f1afd2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:06.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.450+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1af81d7690 con 0x7f1af80a98f0 2024-03-25T18:13:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.494+0000 7f1aeffff640 1 -- 172.21.15.84:0/1849795149 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1af81d7690 con 0x7f1af80a98f0 2024-03-25T18:13:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.494+0000 7f1aeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.494+0000 7f1aeffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:06.496 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:06.496 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:06.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 10 librados: watch_flush enter 2024-03-25T18:13:06.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 10 librados: watch_flush exit 2024-03-25T18:13:06.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T18:13:06.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 10 client.4580.objecter successfully canceled tick 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1ad806ddd0 legacy=0x7f1ad8070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 10 monclient: shutdown 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 >> v1:172.21.15.84:6790/0 conn(0x7f1af80a98f0 legacy=0x7f1af8137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 shutdown_connections 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 >> 172.21.15.84:0/1849795149 conn(0x7f1af80b7ea0 msgr2=0x7f1af813e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.497+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 shutdown_connections 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.498+0000 7f1afd2d9640 1 -- 172.21.15.84:0/1849795149 wait complete. 2024-03-25T18:13:06.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:06.498+0000 7f1afd2d9640 1 librados: shutdown 2024-03-25T18:13:06.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:13.318 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:13.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.431+0000 7f1a179c6640 10 monclient: get_monmap_and_config 2024-03-25T18:13:13.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.431+0000 7f1a179c6640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.431+0000 7f1a179c6640 10 monclient: monmap: 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:13.432766+0000 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:13.432766+0000 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:13.433 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.431+0000 7f1a179c6640 1 Processor -- start 2024-03-25T18:13:13.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 1 -- start start 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient: init 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1a100a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1a101430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a10142520 con 0x7f1a1013f710 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b28a0 con 0x7f1a100a98f0 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b2a30 con 0x7f1a101430c0 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.432+0000 7f1a179c6640 10 monclient(hunting): authenticate will time out at 7437.731708s 2024-03-25T18:13:13.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.475+0000 7f1a15762640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1a100a98f0 0x7f1a100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:58986/0 (socket says 172.21.15.84:58986) 2024-03-25T18:13:13.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.475+0000 7f1a15762640 1 -- 172.21.15.84:0/3508905950 learned_addr learned my addr 172.21.15.84:0/3508905950 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827594547 0 0) 0x7f1a10142520 con 0x7f1a1013f710 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:13:13.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.486+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19fc003520 con 0x7f1a1013f710 2024-03-25T18:13:13.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.490+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224611866 0 0) 0x7f19fc003520 con 0x7f1a1013f710 2024-03-25T18:13:13.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.490+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 >> v1:172.21.15.84:6790/0 conn(0x7f1a101430c0 legacy=0x7f1a1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 >> v1:172.21.15.157:6789/0 conn(0x7f1a100a98f0 legacy=0x7f1a100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 1 monclient: found mon.noname-a 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100b44c0 con 0x7f1a1013f710 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a00001690 con 0x7f1a1013f710 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a07fff640 10 monclient: dump: 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.493 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a179c6640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:13:13.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.491+0000 7f1a179c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a00001860 con 0x7f1a1013f710 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 1 -- 172.21.15.84:0/3508905950 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a00001cb0 con 0x7f1a1013f710 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a179c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a179c6640 4 set_mon_vals no callback set 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.514+0000 7f1a07fff640 10 monclient: dump: 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.516 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.517 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.517 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.517 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.517 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 10 monclient: shutdown 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 1 -- 172.21.15.84:0/3508905950 >> v1:172.21.15.84:6789/0 conn(0x7f1a1013f710 legacy=0x7f1a101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 1 -- 172.21.15.84:0/3508905950 shutdown_connections 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 1 -- 172.21.15.84:0/3508905950 >> 172.21.15.84:0/3508905950 conn(0x7f1a100b7ea0 msgr2=0x7f1a100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 1 -- 172.21.15.84:0/3508905950 shutdown_connections 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.515+0000 7f1a179c6640 1 -- 172.21.15.84:0/3508905950 wait complete. 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 10 monclient: monmap: 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:13.519 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 librados: starting msgr at 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 librados: starting objecter 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 Processor -- start 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 -- start start 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 librados: setting wanted keys 2024-03-25T18:13:13.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 1 librados: calling monclient init 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.516+0000 7f1a179c6640 10 monclient: init 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1a100a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1a101430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b0870 con 0x7f1a101430c0 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b16c0 con 0x7f1a1013f710 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b0ae0 con 0x7f1a100a98f0 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.517+0000 7f1a179c6640 10 monclient(hunting): authenticate will time out at 7437.816242s 2024-03-25T18:13:13.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.529+0000 7f1a15f63640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1a101430c0 0x7f1a101e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55008/0 (socket says 172.21.15.84:55008) 2024-03-25T18:13:13.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.529+0000 7f1a15f63640 1 -- 172.21.15.84:0/2986309038 learned_addr learned my addr 172.21.15.84:0/2986309038 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:13.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.533+0000 7f1a05ffb640 10 client.?.objecter ms_handle_connect 0x7f1a100a98f0 2024-03-25T18:13:13.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.533+0000 7f1a05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.533+0000 7f1a05ffb640 10 client.?.objecter ms_handle_connect 0x7f1a101430c0 2024-03-25T18:13:13.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.533+0000 7f1a05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.536+0000 7f1a05ffb640 10 client.?.objecter ms_handle_connect 0x7f1a1013f710 2024-03-25T18:13:13.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.536+0000 7f1a05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062483982 0 0) 0x7f1a100b0ae0 con 0x7f1a100a98f0 2024-03-25T18:13:13.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:13:13.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.543+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f4004eb0 con 0x7f1a100a98f0 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582378900 0 0) 0x7f1a100b0870 con 0x7f1a101430c0 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:13:13.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.547+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a100b0ae0 con 0x7f1a101430c0 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349011658 0 0) 0x7f1a100b16c0 con 0x7f1a1013f710 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a100b0870 con 0x7f1a1013f710 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 531222357 0 0) 0x7f1a100b0ae0 con 0x7f1a101430c0 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a100b16c0 con 0x7f1a101430c0 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a08001a50 con 0x7f1a101430c0 2024-03-25T18:13:13.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.564+0000 7f1a05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086201589 0 0) 0x7f19f4004eb0 con 0x7f1a100a98f0 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a100b0ae0 con 0x7f1a100a98f0 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a0c0018a0 con 0x7f1a100a98f0 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.565+0000 7f1a05ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785340173 0 0) 0x7f1a100b0870 con 0x7f1a1013f710 2024-03-25T18:13:13.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f4004eb0 con 0x7f1a1013f710 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a00001b70 con 0x7f1a1013f710 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.582+0000 7f1a05ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.584 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1991192769 0 0) 0x7f1a100b16c0 con 0x7f1a101430c0 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 >> v1:172.21.15.84:6790/0 conn(0x7f1a100a98f0 legacy=0x7f1a100abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 >> v1:172.21.15.157:6789/0 conn(0x7f1a1013f710 legacy=0x7f1a101df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 1 monclient: found mon.a 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100b0dc0 con 0x7f1a101430c0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 10 monclient: _renew_subs 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a100ae760 con 0x7f1a101430c0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 10 monclient: _renew_subs 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a100b1430 con 0x7f1a101430c0 2024-03-25T18:13:13.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.586+0000 7f1a179c6640 1 librados: init done 2024-03-25T18:13:13.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.588+0000 7f1a179c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.588+0000 7f1a179c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.588+0000 7f1a179c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:13.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.588+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a100627b0 con 0x7f1a101430c0 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a08001f60 con 0x7f1a101430c0 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1a08004d80 con 0x7f1a101430c0 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a05ffb640 10 monclient: dump: 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:13.911 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.909+0000 7f1a077fe640 4 set_mon_vals no callback set 2024-03-25T18:13:13.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.910+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1a0801ad50 con 0x7f1a101430c0 2024-03-25T18:13:13.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.911+0000 7f1a15762640 10 monclient: get_auth_request con 0x7f19f406e0b0 auth_method 0 2024-03-25T18:13:13.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.911+0000 7f1a05ffb640 10 client.4593.objecter ms_handle_connect 0x7f19f406e0b0 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1a08054900 con 0x7f1a101430c0 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 10 client.4593.objecter ms_dispatch 0x7f1a10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a08050410 con 0x7f1a101430c0 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:13.916+0000 7f1a05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:14.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.177+0000 7f1a179c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.177+0000 7f1a179c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.177+0000 7f1a179c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:14.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.177+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a10141de0 con 0x7f1a101430c0 2024-03-25T18:13:14.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.211+0000 7f1a05ffb640 1 -- 172.21.15.84:0/2986309038 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1a08052340 con 0x7f1a101430c0 2024-03-25T18:13:14.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.211+0000 7f1a05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.211+0000 7f1a05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:14.213 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:14.213 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 10 librados: watch_flush enter 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 10 librados: watch_flush exit 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 10 client.4593.objecter successfully canceled tick 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 >> v1:172.21.15.157:6800/1801259026 conn(0x7f19f406e0b0 legacy=0x7f19f4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 10 monclient: shutdown 2024-03-25T18:13:14.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 >> v1:172.21.15.84:6789/0 conn(0x7f1a101430c0 legacy=0x7f1a101e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 shutdown_connections 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.214+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 >> 172.21.15.84:0/2986309038 conn(0x7f1a100b7ea0 msgr2=0x7f1a10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.215+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 shutdown_connections 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.215+0000 7f1a179c6640 1 -- 172.21.15.84:0/2986309038 wait complete. 2024-03-25T18:13:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:14.215+0000 7f1a179c6640 1 librados: shutdown 2024-03-25T18:13:14.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:21.035 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 10 monclient: get_monmap_and_config 2024-03-25T18:13:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 10 monclient: monmap: 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:21.135555+0000 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:21.135555+0000 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:21.136 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 1 Processor -- start 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 1 -- start start 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.134+0000 7f6bf8625640 10 monclient: init 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf00b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00af940 con 0x7f6bf00b4d60 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b0a70 con 0x7f6bf00b1730 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b0d50 con 0x7f6bf014be90 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.135+0000 7f6bf8625640 10 monclient(hunting): authenticate will time out at 7445.434782s 2024-03-25T18:13:21.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.145+0000 7f6bf63c1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6bf00b4d60 0x7f6bf013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:50702/0 (socket says 172.21.15.84:50702) 2024-03-25T18:13:21.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.145+0000 7f6bf63c1640 1 -- 172.21.15.84:0/3592792221 learned_addr learned my addr 172.21.15.84:0/3592792221 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008013444 0 0) 0x7f6bf00af940 con 0x7f6bf00b4d60 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:13:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.153+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc003520 con 0x7f6bf00b4d60 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408337700 0 0) 0x7f6bf00b0d50 con 0x7f6bf014be90 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:13:21.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.157+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf00af940 con 0x7f6bf014be90 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798581608 0 0) 0x7f6bdc003520 con 0x7f6bf00b4d60 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 >> v1:172.21.15.84:6790/0 conn(0x7f6bf014be90 legacy=0x7f6bf014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 >> v1:172.21.15.157:6789/0 conn(0x7f6bf00b1730 legacy=0x7f6bf00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 monclient: found mon.noname-a 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf00b1030 con 0x7f6bf00b4d60 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6bec001690 con 0x7f6bf00b4d60 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf53bf640 10 monclient: dump: 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.170 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf8625640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:13:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.167+0000 7f6bf8625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.175+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bec0017f0 con 0x7f6bf00b4d60 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.175+0000 7f6bf53bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.175+0000 7f6bf53bf640 1 -- 172.21.15.84:0/3592792221 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6bec001c00 con 0x7f6bf00b4d60 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.175+0000 7f6bf8625640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf8625640 4 set_mon_vals no callback set 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf53bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf53bf640 10 monclient: dump: 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:21.178 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf8625640 10 monclient: shutdown 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf8625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.176+0000 7f6bf8625640 1 -- 172.21.15.84:0/3592792221 >> v1:172.21.15.84:6789/0 conn(0x7f6bf00b4d60 legacy=0x7f6bf013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.177+0000 7f6bf8625640 1 -- 172.21.15.84:0/3592792221 shutdown_connections 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.177+0000 7f6bf8625640 1 -- 172.21.15.84:0/3592792221 >> 172.21.15.84:0/3592792221 conn(0x7f6bf00617a0 msgr2=0x7f6bf0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.177+0000 7f6bf8625640 1 -- 172.21.15.84:0/3592792221 shutdown_connections 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.177+0000 7f6bf8625640 1 -- 172.21.15.84:0/3592792221 wait complete. 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.177+0000 7f6bf8625640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient: monmap: 2024-03-25T18:13:21.181 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 librados: starting msgr at 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 librados: starting objecter 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 Processor -- start 2024-03-25T18:13:21.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 -- start start 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 librados: setting wanted keys 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 1 librados: calling monclient init 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient: init 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.178+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01368f0 con 0x7f6bf00b1730 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf0137750 con 0x7f6bf00b4d60 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01363d0 con 0x7f6bf014be90 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.179+0000 7f6bf8625640 10 monclient(hunting): authenticate will time out at 7445.478001s 2024-03-25T18:13:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.192+0000 7f6bf5bc0640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6bf00b4d60 0x7f6bf012fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:55748/0 (socket says 172.21.15.84:55748) 2024-03-25T18:13:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.192+0000 7f6bf5bc0640 1 -- 172.21.15.84:0/1516322616 learned_addr learned my addr 172.21.15.84:0/1516322616 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:21.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.193+0000 7f6be6ffd640 10 client.?.objecter ms_handle_connect 0x7f6bf00b4d60 2024-03-25T18:13:21.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.193+0000 7f6be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.202+0000 7f6be6ffd640 10 client.?.objecter ms_handle_connect 0x7f6bf014be90 2024-03-25T18:13:21.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.202+0000 7f6be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807294783 0 0) 0x7f6bf01363d0 con 0x7f6bf014be90 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:13:21.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.212+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd4004eb0 con 0x7f6bf014be90 2024-03-25T18:13:21.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 client.?.objecter ms_handle_connect 0x7f6bf00b1730 2024-03-25T18:13:21.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947111540 0 0) 0x7f6bf0137750 con 0x7f6bf00b4d60 2024-03-25T18:13:21.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:13:21.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.215+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf01363d0 con 0x7f6bf00b4d60 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127899274 0 0) 0x7f6bf01368f0 con 0x7f6bf00b1730 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:13:21.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.219+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0137750 con 0x7f6bf00b1730 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843731715 0 0) 0x7f6bd4004eb0 con 0x7f6bf014be90 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bf01368f0 con 0x7f6bf014be90 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6be8001bd0 con 0x7f6bf014be90 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:21.224 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.222+0000 7f6be6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339398058 0 0) 0x7f6bf01363d0 con 0x7f6bf00b4d60 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bd4004eb0 con 0x7f6bf00b4d60 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6be00019b0 con 0x7f6bf00b4d60 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.227+0000 7f6be6ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:21.229 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:21.230 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2488728619 0 0) 0x7f6bf01368f0 con 0x7f6bf014be90 2024-03-25T18:13:21.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 >> v1:172.21.15.157:6789/0 conn(0x7f6bf00b4d60 legacy=0x7f6bf012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 >> v1:172.21.15.84:6789/0 conn(0x7f6bf00b1730 legacy=0x7f6bf0135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 1 monclient: found mon.c 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0130470 con 0x7f6bf014be90 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 10 monclient: _renew_subs 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf0130720 con 0x7f6bf014be90 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 10 monclient: _renew_subs 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:21.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf01322f0 con 0x7f6bf014be90 2024-03-25T18:13:21.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.229+0000 7f6bf8625640 1 librados: init done 2024-03-25T18:13:21.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.230+0000 7f6bf8625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.230+0000 7f6bf8625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.230+0000 7f6bf8625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:21.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.230+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf0136210 con 0x7f6bf014be90 2024-03-25T18:13:21.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6be80020c0 con 0x7f6bf014be90 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f6be8003b10 con 0x7f6bf014be90 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6be6ffd640 10 monclient: dump: 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:21.538 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.536+0000 7f6bf4bbe640 4 set_mon_vals no callback set 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.537+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f6be8019ae0 con 0x7f6bf014be90 2024-03-25T18:13:21.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.537+0000 7f6bf63c1640 10 monclient: get_auth_request con 0x7f6bd406df20 auth_method 0 2024-03-25T18:13:21.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.538+0000 7f6be6ffd640 10 client.4589.objecter ms_handle_connect 0x7f6bd406df20 2024-03-25T18:13:21.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f6be8053780 con 0x7f6bf014be90 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 10 client.4589.objecter ms_dispatch 0x7f6bf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6be804f0c0 con 0x7f6bf014be90 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.543+0000 7f6be6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:21.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.813+0000 7f6bf8625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.813+0000 7f6bf8625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.813+0000 7f6bf8625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.813+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bf00b2270 con 0x7f6bf014be90 2024-03-25T18:13:21.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.834+0000 7f6be6ffd640 1 -- 172.21.15.84:0/1516322616 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f6be80511c0 con 0x7f6bf014be90 2024-03-25T18:13:21.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.834+0000 7f6be6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:21.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.834+0000 7f6be6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:21.836 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:21.837 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.837+0000 7f6bf8625640 10 librados: watch_flush enter 2024-03-25T18:13:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.837+0000 7f6bf8625640 10 librados: watch_flush exit 2024-03-25T18:13:21.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.837+0000 7f6bf8625640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.837+0000 7f6bf8625640 10 client.4589.objecter successfully canceled tick 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.837+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 >> v1:172.21.15.157:6800/1801259026 conn(0x7f6bd406df20 legacy=0x7f6bd40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 10 monclient: shutdown 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 >> v1:172.21.15.84:6790/0 conn(0x7f6bf014be90 legacy=0x7f6bf01d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 shutdown_connections 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 >> 172.21.15.84:0/1516322616 conn(0x7f6bf00617a0 msgr2=0x7f6bf0138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 shutdown_connections 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 -- 172.21.15.84:0/1516322616 wait complete. 2024-03-25T18:13:21.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:21.838+0000 7f6bf8625640 1 librados: shutdown 2024-03-25T18:13:21.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:23.951 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:20.770 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:23.959 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:20.778 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:23.995 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:20.779 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.002 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:20.822 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.004 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:20.823 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.010 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:20.794 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.014 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:20.833 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.044 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:20.863 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.048 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:20.832 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:28.657 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.772+0000 7f1b57354640 10 monclient: get_monmap_and_config 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.772+0000 7f1b57354640 10 monclient: build_initial_monmap 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.772+0000 7f1b57354640 10 monclient: monmap: 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:28.774135+0000 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:28.774135+0000 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:28.774 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:28.775 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:28.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.773+0000 7f1b57354640 1 Processor -- start 2024-03-25T18:13:28.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.773+0000 7f1b57354640 1 -- start start 2024-03-25T18:13:28.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.773+0000 7f1b57354640 10 monclient: init 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-c con 0x7f1b500b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-b con 0x7f1b500b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-a con 0x7f1b5014be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500af940 con 0x7f1b5014be90 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500b0a70 con 0x7f1b500b4d60 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500b0d50 con 0x7f1b500b1730 2024-03-25T18:13:28.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:28.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.774+0000 7f1b57354640 10 monclient(hunting): authenticate will time out at 7453.073663s 2024-03-25T18:13:28.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.775+0000 7f1b550f0640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1b500b4d60 0x7f1b5013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51654/0 (socket says 172.21.15.84:51654) 2024-03-25T18:13:28.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.775+0000 7f1b550f0640 1 -- 172.21.15.84:0/3587062172 learned_addr learned my addr 172.21.15.84:0/3587062172 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821284406 0 0) 0x7f1b500b0a70 con 0x7f1b500b4d60 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:13:28.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.784+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3c003520 con 0x7f1b500b4d60 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100133908 0 0) 0x7f1b500af940 con 0x7f1b5014be90 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:13:28.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.791+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b500b0a70 con 0x7f1b5014be90 2024-03-25T18:13:28.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3209073218 0 0) 0x7f1b3c003520 con 0x7f1b500b4d60 2024-03-25T18:13:28.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 >> v1:172.21.15.84:6790/0 conn(0x7f1b500b1730 legacy=0x7f1b500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:28.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 >> v1:172.21.15.84:6789/0 conn(0x7f1b5014be90 legacy=0x7f1b5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 1 monclient: found mon.noname-c 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b500b1030 con 0x7f1b500b4d60 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.797+0000 7f1b47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1b4c001690 con 0x7f1b500b4d60 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b47fff640 10 monclient: dump: 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.800 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b57354640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:13:28.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.798+0000 7f1b57354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b4c0017f0 con 0x7f1b500b4d60 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 1 -- 172.21.15.84:0/3587062172 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1b4c001c00 con 0x7f1b500b4d60 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b47fff640 10 monclient: dump: 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:28.813 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:28.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 4 set_mon_vals no callback set 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 10 monclient: shutdown 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 1 -- 172.21.15.84:0/3587062172 >> v1:172.21.15.157:6789/0 conn(0x7f1b500b4d60 legacy=0x7f1b5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 1 -- 172.21.15.84:0/3587062172 shutdown_connections 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 1 -- 172.21.15.84:0/3587062172 >> 172.21.15.84:0/3587062172 conn(0x7f1b500617a0 msgr2=0x7f1b50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.812+0000 7f1b57354640 1 -- 172.21.15.84:0/3587062172 shutdown_connections 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.813+0000 7f1b57354640 1 -- 172.21.15.84:0/3587062172 wait complete. 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.813+0000 7f1b57354640 10 monclient: build_initial_monmap 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.813+0000 7f1b57354640 10 monclient: monmap: 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:28.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 librados: starting msgr at 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 librados: starting objecter 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 Processor -- start 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 -- start start 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 librados: setting wanted keys 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 1 librados: calling monclient init 2024-03-25T18:13:28.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.814+0000 7f1b57354640 10 monclient: init 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-c con 0x7f1b500b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-b con 0x7f1b500b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient(hunting): picked mon.noname-a con 0x7f1b5014be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b50138840 con 0x7f1b5014be90 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b501d8620 con 0x7f1b500b4d60 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b501d79f0 con 0x7f1b500b1730 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:28.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.815+0000 7f1b57354640 10 monclient(hunting): authenticate will time out at 7453.114489s 2024-03-25T18:13:28.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.819+0000 7f1b548ef640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1b500b4d60 0x7f1b50137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51662/0 (socket says 172.21.15.84:51662) 2024-03-25T18:13:28.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.819+0000 7f1b548ef640 1 -- 172.21.15.84:0/1876032836 learned_addr learned my addr 172.21.15.84:0/1876032836 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:28.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.820+0000 7f1b45ffb640 10 client.?.objecter ms_handle_connect 0x7f1b500b4d60 2024-03-25T18:13:28.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.820+0000 7f1b45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864603742 0 0) 0x7f1b501d8620 con 0x7f1b500b4d60 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:13:28.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.829+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b34004eb0 con 0x7f1b500b4d60 2024-03-25T18:13:28.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.831+0000 7f1b45ffb640 10 client.?.objecter ms_handle_connect 0x7f1b500b1730 2024-03-25T18:13:28.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.831+0000 7f1b45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:28.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.834+0000 7f1b45ffb640 10 client.?.objecter ms_handle_connect 0x7f1b5014be90 2024-03-25T18:13:28.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.834+0000 7f1b45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:28.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716201192 0 0) 0x7f1b50138840 con 0x7f1b5014be90 2024-03-25T18:13:28.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:13:28.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.841+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b501d8620 con 0x7f1b5014be90 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135165671 0 0) 0x7f1b34004eb0 con 0x7f1b500b4d60 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b50138840 con 0x7f1b500b4d60 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1b40001960 con 0x7f1b500b4d60 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.847 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:28.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.845+0000 7f1b45ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:28.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429867834 0 0) 0x7f1b501d79f0 con 0x7f1b500b1730 2024-03-25T18:13:28.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:13:28.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.850+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b34004eb0 con 0x7f1b500b1730 2024-03-25T18:13:28.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3380172538 0 0) 0x7f1b50138840 con 0x7f1b500b4d60 2024-03-25T18:13:28.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 >> v1:172.21.15.84:6790/0 conn(0x7f1b500b1730 legacy=0x7f1b501d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 >> v1:172.21.15.84:6789/0 conn(0x7f1b5014be90 legacy=0x7f1b50138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 1 monclient: found mon.b 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50142f20 con 0x7f1b500b4d60 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 10 monclient: _renew_subs 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b50143170 con 0x7f1b500b4d60 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 10 monclient: _renew_subs 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:13:28.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b50140560 con 0x7f1b500b4d60 2024-03-25T18:13:28.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.855+0000 7f1b57354640 1 librados: init done 2024-03-25T18:13:28.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.856+0000 7f1b57354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:28.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.856+0000 7f1b57354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:28.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.856+0000 7f1b57354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:13:28.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:28.856+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b50062100 con 0x7f1b500b4d60 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b400035f0 con 0x7f1b500b4d60 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1b40003940 con 0x7f1b500b4d60 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b45ffb640 10 monclient: dump: 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:29.128 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.126+0000 7f1b477fe640 4 set_mon_vals no callback set 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.127+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1b40003b20 con 0x7f1b500b4d60 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.127+0000 7f1b550f0640 10 monclient: get_auth_request con 0x7f1b3406ddb0 auth_method 0 2024-03-25T18:13:29.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.127+0000 7f1b45ffb640 10 client.4600.objecter ms_handle_connect 0x7f1b3406ddb0 2024-03-25T18:13:29.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1b40054820 con 0x7f1b500b4d60 2024-03-25T18:13:29.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 10 client.4600.objecter ms_dispatch 0x7f1b50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:29.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:29.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:29.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:13:29.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b40050330 con 0x7f1b500b4d60 2024-03-25T18:13:29.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.132+0000 7f1b45ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:29.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.390+0000 7f1b57354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.390+0000 7f1b57354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.390+0000 7f1b57354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:13:29.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.390+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b500627b0 con 0x7f1b500b4d60 2024-03-25T18:13:29.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.410+0000 7f1b45ffb640 1 -- 172.21.15.84:0/1876032836 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1b40052260 con 0x7f1b500b4d60 2024-03-25T18:13:29.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.410+0000 7f1b45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.410+0000 7f1b45ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:29.411 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:29.411 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 10 librados: watch_flush enter 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 10 librados: watch_flush exit 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1b3406ddb0 legacy=0x7f1b34070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 10 monclient: shutdown 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 >> v1:172.21.15.157:6789/0 conn(0x7f1b500b4d60 legacy=0x7f1b50137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 shutdown_connections 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 >> 172.21.15.84:0/1876032836 conn(0x7f1b500617a0 msgr2=0x7f1b500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 shutdown_connections 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.412+0000 7f1b57354640 1 -- 172.21.15.84:0/1876032836 wait complete. 2024-03-25T18:13:29.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:29.413+0000 7f1b57354640 1 librados: shutdown 2024-03-25T18:13:29.426 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:36.233 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 10 monclient: monmap: 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:36.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:36.344602+0000 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:36.344602+0000 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:36.345 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 1 Processor -- start 2024-03-25T18:13:36.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 1 -- start start 2024-03-25T18:13:36.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.343+0000 7f9df861e640 10 monclient: init 2024-03-25T18:13:36.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-a con 0x7f9df00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-b con 0x7f9df013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-c con 0x7f9df01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0142520 con 0x7f9df00a98f0 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00b28a0 con 0x7f9df01430c0 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00b2a30 con 0x7f9df013f710 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.344+0000 7f9df861e640 10 monclient(hunting): authenticate will time out at 7460.643675s 2024-03-25T18:13:36.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.346+0000 7f9df6bbb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9df01430c0 0x7f9df014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56420/0 (socket says 172.21.15.84:56420) 2024-03-25T18:13:36.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.346+0000 7f9df6bbb640 1 -- 172.21.15.84:0/3467485374 learned_addr learned my addr 172.21.15.84:0/3467485374 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471753533 0 0) 0x7f9df00b28a0 con 0x7f9df01430c0 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:13:36.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.351+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc003520 con 0x7f9df01430c0 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2399950707 0 0) 0x7f9ddc003520 con 0x7f9df01430c0 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 >> v1:172.21.15.84:6790/0 conn(0x7f9df013f710 legacy=0x7f9df01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 >> v1:172.21.15.84:6789/0 conn(0x7f9df00a98f0 legacy=0x7f9df00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 monclient: found mon.noname-c 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:13:36.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00b44c0 con 0x7f9df01430c0 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9de8001a10 con 0x7f9df01430c0 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df861e640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df861e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df53b8640 10 monclient: dump: 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.363 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:36.364 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:36.364 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:36.364 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:36.364 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.360+0000 7f9df861e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de8001be0 con 0x7f9df01430c0 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 1 -- 172.21.15.84:0/3467485374 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9de8002030 con 0x7f9df01430c0 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df861e640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df861e640 4 set_mon_vals no callback set 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.373+0000 7f9df53b8640 10 monclient: dump: 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.375 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:36.376 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 10 monclient: shutdown 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 1 -- 172.21.15.84:0/3467485374 >> v1:172.21.15.157:6789/0 conn(0x7f9df01430c0 legacy=0x7f9df014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 1 -- 172.21.15.84:0/3467485374 shutdown_connections 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 1 -- 172.21.15.84:0/3467485374 >> 172.21.15.84:0/3467485374 conn(0x7f9df00b7ea0 msgr2=0x7f9df00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 1 -- 172.21.15.84:0/3467485374 shutdown_connections 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.374+0000 7f9df861e640 1 -- 172.21.15.84:0/3467485374 wait complete. 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 10 monclient: monmap: 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 librados: starting msgr at 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 librados: starting objecter 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 Processor -- start 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 -- start start 2024-03-25T18:13:36.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 librados: setting wanted keys 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 1 librados: calling monclient init 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.375+0000 7f9df861e640 10 monclient: init 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-b con 0x7f9df00a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-a con 0x7f9df013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient(hunting): picked mon.noname-c con 0x7f9df01430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df01384b0 con 0x7f9df013f710 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df01394e0 con 0x7f9df00a98f0 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0139750 con 0x7f9df01430c0 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.376+0000 7f9df861e640 10 monclient(hunting): authenticate will time out at 7460.675479s 2024-03-25T18:13:36.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.394+0000 7f9df5bb9640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f9df013f710 0x7f9df0134de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:38808/0 (socket says 172.21.15.84:38808) 2024-03-25T18:13:36.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.394+0000 7f9df5bb9640 1 -- 172.21.15.84:0/1538939718 learned_addr learned my addr 172.21.15.84:0/1538939718 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:36.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.395+0000 7f9de6ffd640 10 client.?.objecter ms_handle_connect 0x7f9df013f710 2024-03-25T18:13:36.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.395+0000 7f9de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.399+0000 7f9de6ffd640 10 client.?.objecter ms_handle_connect 0x7f9df01430c0 2024-03-25T18:13:36.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.399+0000 7f9de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622241706 0 0) 0x7f9df01384b0 con 0x7f9df013f710 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:13:36.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.400+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4004eb0 con 0x7f9df013f710 2024-03-25T18:13:36.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.402+0000 7f9de6ffd640 10 client.?.objecter ms_handle_connect 0x7f9df00a98f0 2024-03-25T18:13:36.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.402+0000 7f9de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772496324 0 0) 0x7f9df0139750 con 0x7f9df01430c0 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:13:36.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.409+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df01384b0 con 0x7f9df01430c0 2024-03-25T18:13:36.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053288792 0 0) 0x7f9dd4004eb0 con 0x7f9df013f710 2024-03-25T18:13:36.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9df0139750 con 0x7f9df013f710 2024-03-25T18:13:36.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9de0001960 con 0x7f9df013f710 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:36.415 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.413+0000 7f9de6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647775417 0 0) 0x7f9df01394e0 con 0x7f9df00a98f0 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:13:36.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.414+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4004eb0 con 0x7f9df00a98f0 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2480104855 0 0) 0x7f9df0139750 con 0x7f9df013f710 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 >> v1:172.21.15.84:6790/0 conn(0x7f9df01430c0 legacy=0x7f9df01e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 >> v1:172.21.15.157:6789/0 conn(0x7f9df00a98f0 legacy=0x7f9df0137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 1 monclient: found mon.a 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:36.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0138940 con 0x7f9df013f710 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9de6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 10 monclient: _renew_subs 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df0137e70 con 0x7f9df013f710 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 10 monclient: _renew_subs 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df01356d0 con 0x7f9df013f710 2024-03-25T18:13:36.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.423+0000 7f9df861e640 1 librados: init done 2024-03-25T18:13:36.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.425+0000 7f9df861e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.425+0000 7f9df861e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.425+0000 7f9df861e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:36.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.425+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df00627b0 con 0x7f9df013f710 2024-03-25T18:13:36.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de00035f0 con 0x7f9df013f710 2024-03-25T18:13:36.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9de0003940 con 0x7f9df013f710 2024-03-25T18:13:36.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9de6ffd640 10 monclient: dump: 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:36.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.729+0000 7f9df4bb7640 4 set_mon_vals no callback set 2024-03-25T18:13:36.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.730+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f9de0003b60 con 0x7f9df013f710 2024-03-25T18:13:36.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.731+0000 7f9df63ba640 10 monclient: get_auth_request con 0x7f9dd406de00 auth_method 0 2024-03-25T18:13:36.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.731+0000 7f9de6ffd640 10 client.4608.objecter ms_handle_connect 0x7f9dd406de00 2024-03-25T18:13:36.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f9de00547a0 con 0x7f9df013f710 2024-03-25T18:13:36.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 10 client.4608.objecter ms_dispatch 0x7f9df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:36.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:36.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:36.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T18:13:36.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9de00502b0 con 0x7f9df013f710 2024-03-25T18:13:36.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.737+0000 7f9de6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:36.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.984+0000 7f9df861e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:36.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.984+0000 7f9df861e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:36.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.984+0000 7f9df861e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:36.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:36.984+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9df0138660 con 0x7f9df013f710 2024-03-25T18:13:37.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.019+0000 7f9de6ffd640 1 -- 172.21.15.84:0/1538939718 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f9df0138660 con 0x7f9df013f710 2024-03-25T18:13:37.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.019+0000 7f9de6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.019+0000 7f9de6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:37.021 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:37.021 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:37.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 10 librados: watch_flush enter 2024-03-25T18:13:37.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 10 librados: watch_flush exit 2024-03-25T18:13:37.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T18:13:37.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 10 client.4608.objecter successfully canceled tick 2024-03-25T18:13:37.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 >> v1:172.21.15.157:6800/1801259026 conn(0x7f9dd406de00 legacy=0x7f9dd40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 10 monclient: shutdown 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 >> v1:172.21.15.84:6789/0 conn(0x7f9df013f710 legacy=0x7f9df0134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 shutdown_connections 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 >> 172.21.15.84:0/1538939718 conn(0x7f9df00b7ea0 msgr2=0x7f9df0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 shutdown_connections 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.021+0000 7f9df861e640 1 -- 172.21.15.84:0/1538939718 wait complete. 2024-03-25T18:13:37.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:37.022+0000 7f9df861e640 1 librados: shutdown 2024-03-25T18:13:37.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:43.776 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:40.595 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.821 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:40.640 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.825 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:40.644 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.828 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:40.647 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.841 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:43.866 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:08:40.685 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.869 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:08:40.688 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.870 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:40.653 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.877 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:40.661 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.903 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:08:40.687 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:43.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.952+0000 7f93de881640 10 monclient: get_monmap_and_config 2024-03-25T18:13:43.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.952+0000 7f93de881640 10 monclient: build_initial_monmap 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.952+0000 7f93de881640 10 monclient: monmap: 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:43.953776+0000 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:43.953776+0000 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:43.954 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 1 Processor -- start 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 1 -- start start 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient: init 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient(hunting): picked mon.noname-a con 0x7f93d80b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient(hunting): picked mon.noname-c con 0x7f93d80b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient(hunting): picked mon.noname-b con 0x7f93d814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80af940 con 0x7f93d80b4d60 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80b0a70 con 0x7f93d80b1730 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80b0d50 con 0x7f93d814be90 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:43.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.953+0000 7f93de881640 10 monclient(hunting): authenticate will time out at 7468.252791s 2024-03-25T18:13:43.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.985+0000 7f93d7fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f93d80b4d60 0x7f93d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58466/0 (socket says 172.21.15.84:58466) 2024-03-25T18:13:43.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:43.985+0000 7f93d7fff640 1 -- 172.21.15.84:0/526176198 learned_addr learned my addr 172.21.15.84:0/526176198 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:44.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234201071 0 0) 0x7f93d80b0a70 con 0x7f93d80b1730 2024-03-25T18:13:44.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:13:44.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.007+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0003520 con 0x7f93d80b1730 2024-03-25T18:13:44.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146592469 0 0) 0x7f93d80af940 con 0x7f93d80b4d60 2024-03-25T18:13:44.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:13:44.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.011+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d80b0a70 con 0x7f93d80b4d60 2024-03-25T18:13:44.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213695608 0 0) 0x7f93d80b0d50 con 0x7f93d814be90 2024-03-25T18:13:44.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:13:44.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.040+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d80af940 con 0x7f93d814be90 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2515516369 0 0) 0x7f93d80af940 con 0x7f93d814be90 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 >> v1:172.21.15.157:6789/0 conn(0x7f93d80b1730 legacy=0x7f93d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 >> v1:172.21.15.84:6789/0 conn(0x7f93d80b4d60 legacy=0x7f93d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d80b1030 con 0x7f93d814be90 2024-03-25T18:13:44.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f93d0001930 con 0x7f93d814be90 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93de881640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93de881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93d6ffd640 10 monclient: dump: 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:44.069 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:44.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.066+0000 7f93de881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d0001a90 con 0x7f93d814be90 2024-03-25T18:13:44.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 1 -- 172.21.15.84:0/526176198 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f93d0001ee0 con 0x7f93d814be90 2024-03-25T18:13:44.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:44.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 4 set_mon_vals no callback set 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93d6ffd640 10 monclient: dump: 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:44.102 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 10 monclient: shutdown 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 1 -- 172.21.15.84:0/526176198 >> v1:172.21.15.84:6790/0 conn(0x7f93d814be90 legacy=0x7f93d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 1 -- 172.21.15.84:0/526176198 shutdown_connections 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 1 -- 172.21.15.84:0/526176198 >> 172.21.15.84:0/526176198 conn(0x7f93d80617a0 msgr2=0x7f93d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.100+0000 7f93de881640 1 -- 172.21.15.84:0/526176198 shutdown_connections 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.101+0000 7f93de881640 1 -- 172.21.15.84:0/526176198 wait complete. 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.101+0000 7f93de881640 10 monclient: build_initial_monmap 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.101+0000 7f93de881640 10 monclient: monmap: 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:44.104 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.101+0000 7f93de881640 1 librados: starting msgr at 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.101+0000 7f93de881640 1 librados: starting objecter 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.102+0000 7f93de881640 1 Processor -- start 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.102+0000 7f93de881640 1 -- start start 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.102+0000 7f93de881640 1 librados: setting wanted keys 2024-03-25T18:13:44.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.102+0000 7f93de881640 1 librados: calling monclient init 2024-03-25T18:13:44.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.102+0000 7f93de881640 10 monclient: init 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient(hunting): picked mon.noname-b con 0x7f93d80b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient(hunting): picked mon.noname-c con 0x7f93d80b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient(hunting): picked mon.noname-a con 0x7f93d814be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8140ce0 con 0x7f93d814be90 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d81382d0 con 0x7f93d80b1730 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8138460 con 0x7f93d80b4d60 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:44.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.103+0000 7f93de881640 10 monclient(hunting): authenticate will time out at 7468.402188s 2024-03-25T18:13:44.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.112+0000 7f93d77fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f93d80b4d60 0x7f93d8131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46872/0 (socket says 172.21.15.84:46872) 2024-03-25T18:13:44.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.112+0000 7f93d77fe640 1 -- 172.21.15.84:0/3958072194 learned_addr learned my addr 172.21.15.84:0/3958072194 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:44.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.114+0000 7f93d4ff9640 10 client.?.objecter ms_handle_connect 0x7f93d80b4d60 2024-03-25T18:13:44.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.114+0000 7f93d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.117+0000 7f93d4ff9640 10 client.?.objecter ms_handle_connect 0x7f93d80b1730 2024-03-25T18:13:44.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.117+0000 7f93d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195025360 0 0) 0x7f93c0003520 con 0x7f93d80b4d60 2024-03-25T18:13:44.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:13:44.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.124+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4004eb0 con 0x7f93d80b4d60 2024-03-25T18:13:44.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.135+0000 7f93d4ff9640 10 client.?.objecter ms_handle_connect 0x7f93d814be90 2024-03-25T18:13:44.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.135+0000 7f93d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600761126 0 0) 0x7f93d81382d0 con 0x7f93d80b1730 2024-03-25T18:13:44.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:13:44.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.144+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0003520 con 0x7f93d80b1730 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699694333 0 0) 0x7f93c4004eb0 con 0x7f93d80b4d60 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93d81382d0 con 0x7f93d80b4d60 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f93cc001810 con 0x7f93d80b4d60 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.149+0000 7f93d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:44.151 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:44.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.150+0000 7f93d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:44.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231140351 0 0) 0x7f93d8140ce0 con 0x7f93d814be90 2024-03-25T18:13:44.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:13:44.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4004eb0 con 0x7f93d814be90 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.156+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814426225 0 0) 0x7f93d81382d0 con 0x7f93d80b4d60 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 >> v1:172.21.15.157:6789/0 conn(0x7f93d80b1730 legacy=0x7f93d8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 >> v1:172.21.15.84:6789/0 conn(0x7f93d814be90 legacy=0x7f93d8142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 1 monclient: found mon.c 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d81324b0 con 0x7f93d80b4d60 2024-03-25T18:13:44.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 10 monclient: _renew_subs 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d8132760 con 0x7f93d80b4d60 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 10 monclient: _renew_subs 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d8144d80 con 0x7f93d80b4d60 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.157+0000 7f93de881640 1 librados: init done 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.158+0000 7f93de881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.158+0000 7f93de881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.158+0000 7f93de881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.158+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d8140b20 con 0x7f93d80b4d60 2024-03-25T18:13:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93cc001970 con 0x7f93d80b4d60 2024-03-25T18:13:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f93cc0037b0 con 0x7f93d80b4d60 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 10 monclient: dump: 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:44.462 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d67fc640 4 set_mon_vals no callback set 2024-03-25T18:13:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.460+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f93cc0039d0 con 0x7f93d80b4d60 2024-03-25T18:13:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.461+0000 7f93d7fff640 10 monclient: get_auth_request con 0x7f93c406de00 auth_method 0 2024-03-25T18:13:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.461+0000 7f93d4ff9640 10 client.4601.objecter ms_handle_connect 0x7f93c406de00 2024-03-25T18:13:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f93cc054680 con 0x7f93d80b4d60 2024-03-25T18:13:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 10 client.4601.objecter ms_dispatch 0x7f93d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:13:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93cc050190 con 0x7f93d80b4d60 2024-03-25T18:13:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.468+0000 7f93d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:44.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.725+0000 7f93de881640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.725+0000 7f93de881640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.725+0000 7f93de881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:13:44.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.725+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93d80627b0 con 0x7f93d80b4d60 2024-03-25T18:13:44.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.754+0000 7f93d4ff9640 1 -- 172.21.15.84:0/3958072194 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f93cc0520c0 con 0x7f93d80b4d60 2024-03-25T18:13:44.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.754+0000 7f93d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.754+0000 7f93d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:44.755 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:44.755 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 10 librados: watch_flush enter 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 10 librados: watch_flush exit 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 >> v1:172.21.15.157:6800/1801259026 conn(0x7f93c406de00 legacy=0x7f93c40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 10 monclient: shutdown 2024-03-25T18:13:44.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.756+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 >> v1:172.21.15.84:6790/0 conn(0x7f93d80b4d60 legacy=0x7f93d8131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.757+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 shutdown_connections 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.757+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 >> 172.21.15.84:0/3958072194 conn(0x7f93d80617a0 msgr2=0x7f93d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.757+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 shutdown_connections 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.757+0000 7f93de881640 1 -- 172.21.15.84:0/3958072194 wait complete. 2024-03-25T18:13:44.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:44.757+0000 7f93de881640 1 librados: shutdown 2024-03-25T18:13:44.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:51.576 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:51.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.692+0000 7f1bd58c5640 10 monclient: get_monmap_and_config 2024-03-25T18:13:51.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.692+0000 7f1bd58c5640 10 monclient: build_initial_monmap 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.692+0000 7f1bd58c5640 10 monclient: monmap: 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:51.692928+0000 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:51.692928+0000 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:51.693 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 1 Processor -- start 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 1 -- start start 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 10 monclient: init 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.693+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd01430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0142520 con 0x7f1bd00a98f0 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00b28a0 con 0x7f1bd013f710 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00b2a30 con 0x7f1bd01430c0 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:51.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.694+0000 7f1bd58c5640 10 monclient(hunting): authenticate will time out at 7475.992170s 2024-03-25T18:13:51.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.699+0000 7f1bce7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1bd013f710 0x7f1bd01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:43894/0 (socket says 172.21.15.84:43894) 2024-03-25T18:13:51.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.699+0000 7f1bce7fc640 1 -- 172.21.15.84:0/3096690992 learned_addr learned my addr 172.21.15.84:0/3096690992 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477298016 0 0) 0x7f1bd00b28a0 con 0x7f1bd013f710 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:13:51.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.712+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb8003520 con 0x7f1bd013f710 2024-03-25T18:13:51.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810679171 0 0) 0x7f1bd0142520 con 0x7f1bd00a98f0 2024-03-25T18:13:51.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:13:51.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.720+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd00b28a0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2836129315 0 0) 0x7f1bb8003520 con 0x7f1bd013f710 2024-03-25T18:13:51.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 >> v1:172.21.15.84:6790/0 conn(0x7f1bd01430c0 legacy=0x7f1bd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 >> v1:172.21.15.84:6789/0 conn(0x7f1bd00a98f0 legacy=0x7f1bd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 monclient: found mon.noname-c 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00b44c0 con 0x7f1bd013f710 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:51.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1bc4001690 con 0x7f1bd013f710 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bcdffb640 10 monclient: dump: 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:51.731 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bd58c5640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:13:51.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.729+0000 7f1bd58c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:51.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc40017f0 con 0x7f1bd013f710 2024-03-25T18:13:51.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 1 -- 172.21.15.84:0/3096690992 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1bc4001c00 con 0x7f1bd013f710 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bcdffb640 10 monclient: dump: 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:51.748 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:51.749 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:51.749 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:51.749 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bd58c5640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:51.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.747+0000 7f1bd58c5640 4 set_mon_vals no callback set 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 10 monclient: shutdown 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3096690992 >> v1:172.21.15.157:6789/0 conn(0x7f1bd013f710 legacy=0x7f1bd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3096690992 shutdown_connections 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3096690992 >> 172.21.15.84:0/3096690992 conn(0x7f1bd00b7ea0 msgr2=0x7f1bd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:51.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3096690992 shutdown_connections 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.748+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3096690992 wait complete. 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 10 monclient: build_initial_monmap 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 10 monclient: monmap: 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:51.752 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 librados: starting msgr at 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 librados: starting objecter 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 Processor -- start 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 -- start start 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 librados: setting wanted keys 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 1 librados: calling monclient init 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.749+0000 7f1bd58c5640 10 monclient: init 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:51.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00abdc0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd01d7730 con 0x7f1bd01430c0 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0138f10 con 0x7f1bd013f710 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:51.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.750+0000 7f1bd58c5640 10 monclient(hunting): authenticate will time out at 7476.048545s 2024-03-25T18:13:51.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.756+0000 7f1bceffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1bd00a98f0 0x7f1bd0138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49732/0 (socket says 172.21.15.84:49732) 2024-03-25T18:13:51.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.756+0000 7f1bceffd640 1 -- 172.21.15.84:0/3562529157 learned_addr learned my addr 172.21.15.84:0/3562529157 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:51.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.758+0000 7f1bb7fff640 10 client.?.objecter ms_handle_connect 0x7f1bd01430c0 2024-03-25T18:13:51.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.758+0000 7f1bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:51.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.763+0000 7f1bb7fff640 10 client.?.objecter ms_handle_connect 0x7f1bd00a98f0 2024-03-25T18:13:51.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.763+0000 7f1bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:51.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.766+0000 7f1bb7fff640 10 client.?.objecter ms_handle_connect 0x7f1bd013f710 2024-03-25T18:13:51.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.766+0000 7f1bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939435160 0 0) 0x7f1bd01d7730 con 0x7f1bd01430c0 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.767+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc004eb0 con 0x7f1bd01430c0 2024-03-25T18:13:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481125339 0 0) 0x7f1bd00abdc0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:13:51.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.768+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd01d7730 con 0x7f1bd00a98f0 2024-03-25T18:13:51.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691497952 0 0) 0x7f1bd01d7730 con 0x7f1bd00a98f0 2024-03-25T18:13:51.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bd00abdc0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1bc00019f0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 10 monclient(hunting): dump: 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.777+0000 7f1bb7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71710937 0 0) 0x7f1bbc004eb0 con 0x7f1bd01430c0 2024-03-25T18:13:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bd01d7730 con 0x7f1bd01430c0 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1bc8001b90 con 0x7f1bd01430c0 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.779+0000 7f1bb7fff640 10 monclient(hunting): dump: 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:51.780 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852368875 0 0) 0x7f1bd0138f10 con 0x7f1bd013f710 2024-03-25T18:13:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:13:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.782+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc004eb0 con 0x7f1bd013f710 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3175411559 0 0) 0x7f1bd00abdc0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 >> v1:172.21.15.84:6790/0 conn(0x7f1bd013f710 legacy=0x7f1bd01d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 >> v1:172.21.15.157:6789/0 conn(0x7f1bd01430c0 legacy=0x7f1bd01d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 1 monclient: found mon.a 2024-03-25T18:13:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd01390a0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 5 monclient: authenticate success, global_id 4620 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 10 monclient: _renew_subs 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd01d8320 con 0x7f1bd00a98f0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 10 monclient: _renew_subs 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd01d43c0 con 0x7f1bd00a98f0 2024-03-25T18:13:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.785+0000 7f1bd58c5640 1 librados: init done 2024-03-25T18:13:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.786+0000 7f1bd58c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.786+0000 7f1bd58c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.786+0000 7f1bd58c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:51.786+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd00627b0 con 0x7f1bd00a98f0 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc0003610 con 0x7f1bd00a98f0 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1bc00039a0 con 0x7f1bd00a98f0 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bb7fff640 10 monclient: dump: 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:52.083 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:52.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.082+0000 7f1bcd7fa640 4 set_mon_vals no callback set 2024-03-25T18:13:52.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.083+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1bc0003bc0 con 0x7f1bd00a98f0 2024-03-25T18:13:52.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.084+0000 7f1bce7fc640 10 monclient: get_auth_request con 0x7f1bbc06df50 auth_method 0 2024-03-25T18:13:52.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.084+0000 7f1bb7fff640 10 client.4620.objecter ms_handle_connect 0x7f1bbc06df50 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f1bc0054800 con 0x7f1bd00a98f0 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 10 client.4620.objecter ms_dispatch 0x7f1bd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bc0050310 con 0x7f1bd00a98f0 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.090+0000 7f1bb7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:52.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.342+0000 7f1bd58c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.342+0000 7f1bd58c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.342+0000 7f1bd58c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:52.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.342+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bd0069f40 con 0x7f1bd00a98f0 2024-03-25T18:13:52.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.366+0000 7f1bb7fff640 1 -- 172.21.15.84:0/3562529157 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f1bc0052240 con 0x7f1bd00a98f0 2024-03-25T18:13:52.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.366+0000 7f1bb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.366+0000 7f1bb7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:52.367 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:52.368 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 10 librados: watch_flush enter 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 10 librados: watch_flush exit 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 10 client.4620.objecter successfully canceled tick 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1bbc06df50 legacy=0x7f1bbc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 10 monclient: shutdown 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 >> v1:172.21.15.84:6789/0 conn(0x7f1bd00a98f0 legacy=0x7f1bd0138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 shutdown_connections 2024-03-25T18:13:52.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 >> 172.21.15.84:0/3562529157 conn(0x7f1bd00b7ea0 msgr2=0x7f1bd0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:52.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 shutdown_connections 2024-03-25T18:13:52.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 -- 172.21.15.84:0/3562529157 wait complete. 2024-03-25T18:13:52.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:52.369+0000 7f1bd58c5640 1 librados: shutdown 2024-03-25T18:13:52.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:59.187 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:59.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.301+0000 7f3c695c1640 10 monclient: get_monmap_and_config 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.301+0000 7f3c695c1640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.301+0000 7f3c695c1640 10 monclient: monmap: 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:13:59.301726+0000 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:13:59.301726+0000 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:13:59.302 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 1 Processor -- start 2024-03-25T18:13:59.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 1 -- start start 2024-03-25T18:13:59.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient: init 2024-03-25T18:13:59.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-b con 0x7f3c640a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-a con 0x7f3c641430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64142520 con 0x7f3c641430c0 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b28a0 con 0x7f3c6413f710 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b2a30 con 0x7f3c640a98f0 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.302+0000 7f3c695c1640 10 monclient(hunting): authenticate will time out at 7483.600849s 2024-03-25T18:13:59.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.311+0000 7f3c637fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3c641430c0 0x7f3c6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52948/0 (socket says 172.21.15.84:52948) 2024-03-25T18:13:59.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.311+0000 7f3c637fe640 1 -- 172.21.15.84:0/337784426 learned_addr learned my addr 172.21.15.84:0/337784426 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:59.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436547950 0 0) 0x7f3c640b28a0 con 0x7f3c6413f710 2024-03-25T18:13:59.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:13:59.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.322+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c4c003520 con 0x7f3c6413f710 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810801347 0 0) 0x7f3c64142520 con 0x7f3c641430c0 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:13:59.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.324+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c640b28a0 con 0x7f3c641430c0 2024-03-25T18:13:59.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439888650 0 0) 0x7f3c640b2a30 con 0x7f3c640a98f0 2024-03-25T18:13:59.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:13:59.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.331+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64142520 con 0x7f3c640a98f0 2024-03-25T18:13:59.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239825883 0 0) 0x7f3c4c003520 con 0x7f3c6413f710 2024-03-25T18:13:59.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 >> v1:172.21.15.84:6790/0 conn(0x7f3c640a98f0 legacy=0x7f3c640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 >> v1:172.21.15.84:6789/0 conn(0x7f3c641430c0 legacy=0x7f3c6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 monclient: found mon.noname-c 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640b44c0 con 0x7f3c6413f710 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3c58001690 con 0x7f3c6413f710 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c695c1640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c695c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c61ffb640 10 monclient: dump: 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:59.335 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.333+0000 7f3c695c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c580017f0 con 0x7f3c6413f710 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 1 -- 172.21.15.84:0/337784426 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3c58001c00 con 0x7f3c6413f710 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c695c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c695c1640 4 set_mon_vals no callback set 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.343+0000 7f3c61ffb640 10 monclient: dump: 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.344 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:59.345 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:59.345 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:59.345 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:59.345 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 10 monclient: shutdown 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 1 -- 172.21.15.84:0/337784426 >> v1:172.21.15.157:6789/0 conn(0x7f3c6413f710 legacy=0x7f3c641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 1 -- 172.21.15.84:0/337784426 shutdown_connections 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 1 -- 172.21.15.84:0/337784426 >> 172.21.15.84:0/337784426 conn(0x7f3c640b7ea0 msgr2=0x7f3c640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:59.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 1 -- 172.21.15.84:0/337784426 shutdown_connections 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.344+0000 7f3c695c1640 1 -- 172.21.15.84:0/337784426 wait complete. 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.345+0000 7f3c695c1640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.345+0000 7f3c695c1640 10 monclient: monmap: 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.345+0000 7f3c695c1640 1 librados: starting msgr at 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.345+0000 7f3c695c1640 1 librados: starting objecter 2024-03-25T18:13:59.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 Processor -- start 2024-03-25T18:13:59.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 -- start start 2024-03-25T18:13:59.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 librados: setting wanted keys 2024-03-25T18:13:59.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 librados: calling monclient init 2024-03-25T18:13:59.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient: init 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-c con 0x7f3c640a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 10 monclient(hunting): picked mon.noname-a con 0x7f3c641430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c641d77a0 con 0x7f3c641430c0 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.346+0000 7f3c695c1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64138e40 con 0x7f3c6413f710 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.347+0000 7f3c695c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640aba30 con 0x7f3c640a98f0 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.347+0000 7f3c695c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.347+0000 7f3c695c1640 10 monclient(hunting): authenticate will time out at 7483.644913s 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.348+0000 7f3c637fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3c641430c0 0x7f3c64134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52974/0 (socket says 172.21.15.84:52974) 2024-03-25T18:13:59.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.348+0000 7f3c637fe640 1 -- 172.21.15.84:0/2606947079 learned_addr learned my addr 172.21.15.84:0/2606947079 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:13:59.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.349+0000 7f3c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c641430c0 2024-03-25T18:13:59.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.349+0000 7f3c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.357+0000 7f3c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c6413f710 2024-03-25T18:13:59.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.357+0000 7f3c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551329818 0 0) 0x7f3c641d77a0 con 0x7f3c641430c0 2024-03-25T18:13:59.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:13:59.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.358+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50004eb0 con 0x7f3c641430c0 2024-03-25T18:13:59.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.360+0000 7f3c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c640a98f0 2024-03-25T18:13:59.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.360+0000 7f3c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158719577 0 0) 0x7f3c640aba30 con 0x7f3c640a98f0 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:13:59.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.364+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c641d77a0 con 0x7f3c640a98f0 2024-03-25T18:13:59.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1354689771 0 0) 0x7f3c50004eb0 con 0x7f3c641430c0 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c640aba30 con 0x7f3c641430c0 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3c5c001c70 con 0x7f3c641430c0 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.366+0000 7f3c4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 510515071 0 0) 0x7f3c640aba30 con 0x7f3c641430c0 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 >> v1:172.21.15.84:6790/0 conn(0x7f3c640a98f0 legacy=0x7f3c64138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 >> v1:172.21.15.157:6789/0 conn(0x7f3c6413f710 legacy=0x7f3c641d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 1 monclient: found mon.a 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:59.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c641d8340 con 0x7f3c641430c0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 10 monclient: _renew_subs 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64134820 con 0x7f3c641430c0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 10 monclient: _renew_subs 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c641380c0 con 0x7f3c641430c0 2024-03-25T18:13:59.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.377+0000 7f3c695c1640 1 librados: init done 2024-03-25T18:13:59.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.378+0000 7f3c695c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.378+0000 7f3c695c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.378+0000 7f3c695c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:59.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.378+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c640627b0 con 0x7f3c641430c0 2024-03-25T18:13:59.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.594+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c004b60 con 0x7f3c641430c0 2024-03-25T18:13:59.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.594+0000 7f3c4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:59.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.594+0000 7f3c617fa640 4 set_mon_vals no callback set 2024-03-25T18:13:59.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.672+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f3c5c004e70 con 0x7f3c641430c0 2024-03-25T18:13:59.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.672+0000 7f3c4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.672+0000 7f3c4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.672+0000 7f3c4b7fe640 10 monclient: dump: 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:13:59.673 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:13:59.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.673+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f3c5c01ae40 con 0x7f3c641430c0 2024-03-25T18:13:59.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.673+0000 7f3c62ffd640 10 monclient: get_auth_request con 0x7f3c5006dd40 auth_method 0 2024-03-25T18:13:59.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.674+0000 7f3c4b7fe640 10 client.4626.objecter ms_handle_connect 0x7f3c5006dd40 2024-03-25T18:13:59.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f3c5c054b00 con 0x7f3c641430c0 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 10 client.4626.objecter ms_dispatch 0x7f3c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c5c050440 con 0x7f3c641430c0 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.680+0000 7f3c4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:59.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.930+0000 7f3c695c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:59.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.930+0000 7f3c695c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:59.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.930+0000 7f3c695c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:13:59.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.930+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c6414e8b0 con 0x7f3c641430c0 2024-03-25T18:13:59.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.961+0000 7f3c4b7fe640 1 -- 172.21.15.84:0/2606947079 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f3c5c052540 con 0x7f3c641430c0 2024-03-25T18:13:59.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.961+0000 7f3c4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:59.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.961+0000 7f3c4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:59.962 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:13:59.962 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:59.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.963+0000 7f3c695c1640 10 librados: watch_flush enter 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.963+0000 7f3c695c1640 10 librados: watch_flush exit 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.963+0000 7f3c695c1640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.963+0000 7f3c695c1640 10 client.4626.objecter successfully canceled tick 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 >> v1:172.21.15.157:6800/1801259026 conn(0x7f3c5006dd40 legacy=0x7f3c50070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 10 monclient: shutdown 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 >> v1:172.21.15.84:6789/0 conn(0x7f3c641430c0 legacy=0x7f3c64134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 shutdown_connections 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 >> 172.21.15.84:0/2606947079 conn(0x7f3c640b7ea0 msgr2=0x7f3c64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 shutdown_connections 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 -- 172.21.15.84:0/2606947079 wait complete. 2024-03-25T18:13:59.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:13:59.964+0000 7f3c695c1640 1 librados: shutdown 2024-03-25T18:13:59.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:03.847 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:00.666 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.901 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:00.720 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.930 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:00.749 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.950 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:00.733 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.953 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:00.737 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.958 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:00.777 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.967 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:00.787 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:03.983 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:00.766 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.001 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:00.820 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:06.783 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:06.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:06.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 10 monclient: build_initial_monmap 2024-03-25T18:14:06.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 10 monclient: monmap: 2024-03-25T18:14:06.901 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:06.901 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:06.901428+0000 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:06.901428+0000 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:06.902 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:06.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 1 Processor -- start 2024-03-25T18:14:06.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 1 -- start start 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.900+0000 7fe952e1e640 10 monclient: init 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fe94c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fe94c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fe94c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c142520 con 0x7fe94c13f710 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0b28a0 con 0x7fe94c0a98f0 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0b2a30 con 0x7fe94c1430c0 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:06.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.901+0000 7fe952e1e640 10 monclient(hunting): authenticate will time out at 7491.200717s 2024-03-25T18:14:06.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.904+0000 7fe9513bb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe94c1430c0 0x7fe94c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51858/0 (socket says 172.21.15.84:51858) 2024-03-25T18:14:06.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.904+0000 7fe9513bb640 1 -- 172.21.15.84:0/3456122464 learned_addr learned my addr 172.21.15.84:0/3456122464 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801573548 0 0) 0x7fe94c142520 con 0x7fe94c13f710 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:14:06.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.911+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe93c003520 con 0x7fe94c13f710 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231000178 0 0) 0x7fe94c0b2a30 con 0x7fe94c1430c0 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:14:06.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.912+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c142520 con 0x7fe94c1430c0 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215229541 0 0) 0x7fe93c003520 con 0x7fe94c13f710 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 >> v1:172.21.15.84:6790/0 conn(0x7fe94c1430c0 legacy=0x7fe94c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 >> v1:172.21.15.157:6789/0 conn(0x7fe94c0a98f0 legacy=0x7fe94c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c0b44c0 con 0x7fe94c13f710 2024-03-25T18:14:06.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe938001690 con 0x7fe94c13f710 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe952e1e640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe952e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe94b7fe640 10 monclient: dump: 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.925 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:06.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.922+0000 7fe952e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:06.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9380017f0 con 0x7fe94c13f710 2024-03-25T18:14:06.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 1 -- 172.21.15.84:0/3456122464 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe938001c00 con 0x7fe94c13f710 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe94b7fe640 10 monclient: dump: 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:06.931 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:06.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe952e1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:06.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe952e1e640 4 set_mon_vals no callback set 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe952e1e640 10 monclient: shutdown 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe952e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.929+0000 7fe952e1e640 1 -- 172.21.15.84:0/3456122464 >> v1:172.21.15.84:6789/0 conn(0x7fe94c13f710 legacy=0x7fe94c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.930+0000 7fe952e1e640 1 -- 172.21.15.84:0/3456122464 shutdown_connections 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.930+0000 7fe952e1e640 1 -- 172.21.15.84:0/3456122464 >> 172.21.15.84:0/3456122464 conn(0x7fe94c0b7ea0 msgr2=0x7fe94c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.930+0000 7fe952e1e640 1 -- 172.21.15.84:0/3456122464 shutdown_connections 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.930+0000 7fe952e1e640 1 -- 172.21.15.84:0/3456122464 wait complete. 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 10 monclient: build_initial_monmap 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 10 monclient: monmap: 2024-03-25T18:14:06.934 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 librados: starting msgr at 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 librados: starting objecter 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 Processor -- start 2024-03-25T18:14:06.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 -- start start 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 librados: setting wanted keys 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 1 librados: calling monclient init 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.931+0000 7fe952e1e640 10 monclient: init 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fe94c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fe94c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fe94c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0abac0 con 0x7fe94c13f710 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0abda0 con 0x7fe94c0a98f0 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c1380e0 con 0x7fe94c1430c0 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:06.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.932+0000 7fe952e1e640 10 monclient(hunting): authenticate will time out at 7491.231421s 2024-03-25T18:14:06.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.945+0000 7fe9513bb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe94c1430c0 0x7fe94c1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51896/0 (socket says 172.21.15.84:51896) 2024-03-25T18:14:06.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.945+0000 7fe9513bb640 1 -- 172.21.15.84:0/1336018935 learned_addr learned my addr 172.21.15.84:0/1336018935 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:06.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.946+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c1430c0 2024-03-25T18:14:06.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.946+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:06.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.948+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c0a98f0 2024-03-25T18:14:06.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.948+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:06.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.948+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c13f710 2024-03-25T18:14:06.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.948+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:06.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282199655 0 0) 0x7fe94c1380e0 con 0x7fe94c1430c0 2024-03-25T18:14:06.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:14:06.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.954+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe930004eb0 con 0x7fe94c1430c0 2024-03-25T18:14:06.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746203110 0 0) 0x7fe94c0abac0 con 0x7fe94c13f710 2024-03-25T18:14:06.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:14:06.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.968+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c1380e0 con 0x7fe94c13f710 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964094001 0 0) 0x7fe94c0abda0 con 0x7fe94c0a98f0 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:14:06.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.969+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c0abac0 con 0x7fe94c0a98f0 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171983277 0 0) 0x7fe930004eb0 con 0x7fe94c1430c0 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe94c0abda0 con 0x7fe94c1430c0 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe944001c10 con 0x7fe94c1430c0 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 10 monclient(hunting): dump: 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:06.977 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:06.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.975+0000 7fe9497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501182149 0 0) 0x7fe94c1380e0 con 0x7fe94c13f710 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe930004eb0 con 0x7fe94c13f710 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe938001b40 con 0x7fe94c13f710 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:06.999+0000 7fe9497fa640 10 monclient(hunting): dump: 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:07.001 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.002 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:07.002 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:07.002 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:07.002 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:07.002 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1382094698 0 0) 0x7fe94c0abda0 con 0x7fe94c1430c0 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 >> v1:172.21.15.157:6789/0 conn(0x7fe94c0a98f0 legacy=0x7fe94c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 >> v1:172.21.15.84:6789/0 conn(0x7fe94c13f710 legacy=0x7fe94c138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 1 monclient: found mon.c 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c137d90 con 0x7fe94c1430c0 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe9497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 10 monclient: _renew_subs 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c139240 con 0x7fe94c1430c0 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 10 monclient: _renew_subs 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c136830 con 0x7fe94c1430c0 2024-03-25T18:14:07.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.002+0000 7fe952e1e640 1 librados: init done 2024-03-25T18:14:07.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.003+0000 7fe952e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.003+0000 7fe952e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.003+0000 7fe952e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:07.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.003+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c136370 con 0x7fe94c1430c0 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe944004b60 con 0x7fe94c1430c0 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fe944004cd0 con 0x7fe94c1430c0 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe9497fa640 10 monclient: dump: 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:07.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.303+0000 7fe94affd640 4 set_mon_vals no callback set 2024-03-25T18:14:07.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.304+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fe94401aca0 con 0x7fe94c1430c0 2024-03-25T18:14:07.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.305+0000 7fe950bba640 10 monclient: get_auth_request con 0x7fe93006ded0 auth_method 0 2024-03-25T18:14:07.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.305+0000 7fe9497fa640 10 client.4619.objecter ms_handle_connect 0x7fe93006ded0 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fe944054990 con 0x7fe94c1430c0 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 10 client.4619.objecter ms_dispatch 0x7fe94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.310+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9440502d0 con 0x7fe94c1430c0 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.311+0000 7fe9497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.311+0000 7fe9497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:07.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.550+0000 7fe952e1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.550+0000 7fe952e1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.550+0000 7fe952e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:07.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.550+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe94c1393d0 con 0x7fe94c1430c0 2024-03-25T18:14:07.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.588+0000 7fe9497fa640 1 -- 172.21.15.84:0/1336018935 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fe9440523d0 con 0x7fe94c1430c0 2024-03-25T18:14:07.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.588+0000 7fe9497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.588+0000 7fe9497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:07.589 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:07.590 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 10 librados: watch_flush enter 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 10 librados: watch_flush exit 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 10 client.4619.objecter successfully canceled tick 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 >> v1:172.21.15.157:6800/1801259026 conn(0x7fe93006ded0 legacy=0x7fe930070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 10 monclient: shutdown 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 >> v1:172.21.15.84:6790/0 conn(0x7fe94c1430c0 legacy=0x7fe94c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 shutdown_connections 2024-03-25T18:14:07.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 >> 172.21.15.84:0/1336018935 conn(0x7fe94c0b7ea0 msgr2=0x7fe94c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 shutdown_connections 2024-03-25T18:14:07.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.590+0000 7fe952e1e640 1 -- 172.21.15.84:0/1336018935 wait complete. 2024-03-25T18:14:07.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:07.591+0000 7fe952e1e640 1 librados: shutdown 2024-03-25T18:14:07.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:14.408 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:14.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.521+0000 7f0340fd0640 10 monclient: get_monmap_and_config 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.521+0000 7f0340fd0640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.521+0000 7f0340fd0640 10 monclient: monmap: 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:14.522890+0000 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:14.522890+0000 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:14.523 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:14.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.522+0000 7f0340fd0640 1 Processor -- start 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.522+0000 7f0340fd0640 1 -- start start 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.522+0000 7f0340fd0640 10 monclient: init 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.522+0000 7f0340fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.522+0000 7f0340fd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f033c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f033c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f033c1430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c142520 con 0x7f033c13f710 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c0b28a0 con 0x7f033c0a98f0 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c0b2a30 con 0x7f033c1430c0 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.523+0000 7f0340fd0640 10 monclient(hunting): authenticate will time out at 7498.821983s 2024-03-25T18:14:14.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.527+0000 7f0333fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f033c13f710 0x7f033c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55336/0 (socket says 172.21.15.84:55336) 2024-03-25T18:14:14.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.527+0000 7f0333fff640 1 -- 172.21.15.84:0/1754245022 learned_addr learned my addr 172.21.15.84:0/1754245022 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:14.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776546606 0 0) 0x7f033c142520 con 0x7f033c13f710 2024-03-25T18:14:14.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:14:14.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.543+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0328003520 con 0x7f033c13f710 2024-03-25T18:14:14.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951145036 0 0) 0x7f033c0b2a30 con 0x7f033c1430c0 2024-03-25T18:14:14.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:14:14.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.552+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c142520 con 0x7f033c1430c0 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714988405 0 0) 0x7f033c0b28a0 con 0x7f033c0a98f0 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:14:14.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.553+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c0b2a30 con 0x7f033c0a98f0 2024-03-25T18:14:14.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1281937472 0 0) 0x7f033c142520 con 0x7f033c1430c0 2024-03-25T18:14:14.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 >> v1:172.21.15.157:6789/0 conn(0x7f033c0a98f0 legacy=0x7f033c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 >> v1:172.21.15.84:6789/0 conn(0x7f033c13f710 legacy=0x7f033c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 monclient: found mon.noname-b 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c0b44c0 con 0x7f033c1430c0 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f03340019f0 con 0x7f033c1430c0 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0339d9b640 10 monclient: dump: 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:14.578 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0340fd0640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:14:14.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.576+0000 7f0340fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0334001b50 con 0x7f033c1430c0 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 1 -- 172.21.15.84:0/1754245022 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0334001f60 con 0x7f033c1430c0 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0339d9b640 10 monclient: dump: 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:14.604 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:14.605 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:14.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0340fd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:14.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.602+0000 7f0340fd0640 4 set_mon_vals no callback set 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 10 monclient: shutdown 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 1 -- 172.21.15.84:0/1754245022 >> v1:172.21.15.84:6790/0 conn(0x7f033c1430c0 legacy=0x7f033c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 1 -- 172.21.15.84:0/1754245022 shutdown_connections 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 1 -- 172.21.15.84:0/1754245022 >> 172.21.15.84:0/1754245022 conn(0x7f033c0b7ea0 msgr2=0x7f033c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 1 -- 172.21.15.84:0/1754245022 shutdown_connections 2024-03-25T18:14:14.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.603+0000 7f0340fd0640 1 -- 172.21.15.84:0/1754245022 wait complete. 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.604+0000 7f0340fd0640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.604+0000 7f0340fd0640 10 monclient: monmap: 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:14.607 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.604+0000 7f0340fd0640 1 librados: starting msgr at 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.604+0000 7f0340fd0640 1 librados: starting objecter 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.604+0000 7f0340fd0640 1 Processor -- start 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 -- start start 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 librados: setting wanted keys 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 librados: calling monclient init 2024-03-25T18:14:14.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient: init 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f033c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f033c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f033c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c0abac0 con 0x7f033c0a98f0 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c1d4310 con 0x7f033c1430c0 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c138f10 con 0x7f033c13f710 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.605+0000 7f0340fd0640 10 monclient(hunting): authenticate will time out at 7498.904713s 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.608+0000 7f033ad9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 0x7f033c1d4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56972/0 (socket says 172.21.15.84:56972) 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.608+0000 7f033ad9d640 1 -- 172.21.15.84:0/3175846231 learned_addr learned my addr 172.21.15.84:0/3175846231 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.609+0000 7f033ad9d640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 legacy=0x7f033c1d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:14:14.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.609+0000 7f033ad9d640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 legacy=0x7f033c1d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:14:14.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.609+0000 7f033ad9d640 1 --1- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 0x7f033c1d4830 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:14:14.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.609+0000 7f033ad9d640 1 --1- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 0x7f033c1d4830 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:14:14.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.609+0000 7f0332ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.157:6789/0 2024-03-25T18:14:14.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.626+0000 7f0332ffd640 10 client.?.objecter ms_handle_connect 0x7f033c13f710 2024-03-25T18:14:14.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.626+0000 7f0332ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.632+0000 7f0332ffd640 10 client.?.objecter ms_handle_connect 0x7f033c0a98f0 2024-03-25T18:14:14.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.632+0000 7f0332ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653775240 0 0) 0x7f033c138f10 con 0x7f033c13f710 2024-03-25T18:14:14.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:14:14.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.641+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c004eb0 con 0x7f033c13f710 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190388293 0 0) 0x7f033c0abac0 con 0x7f033c0a98f0 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:14:14.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.646+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c138f10 con 0x7f033c0a98f0 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490258768 0 0) 0x7f031c004eb0 con 0x7f033c13f710 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f033c0abac0 con 0x7f033c13f710 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0324001950 con 0x7f033c13f710 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:14.669 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:14.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.667+0000 7f0332ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 41631907 0 0) 0x7f033c0abac0 con 0x7f033c13f710 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6789/0 conn(0x7f033c1430c0 legacy=0x7f033c1d4830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.84:6789/0 conn(0x7f033c0a98f0 legacy=0x7f033c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 1 monclient: found mon.c 2024-03-25T18:14:14.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c135010 con 0x7f033c13f710 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 5 monclient: authenticate success, global_id 4625 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 10 monclient: _renew_subs 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c134790 con 0x7f033c13f710 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 10 monclient: _renew_subs 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c1d5120 con 0x7f033c13f710 2024-03-25T18:14:14.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.687+0000 7f0340fd0640 1 librados: init done 2024-03-25T18:14:14.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.689+0000 7f0340fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.689+0000 7f0340fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.689+0000 7f0340fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:14.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:14.689+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c1d44a0 con 0x7f033c13f710 2024-03-25T18:14:15.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0324001d60 con 0x7f033c13f710 2024-03-25T18:14:15.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0324004980 con 0x7f033c13f710 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f0332ffd640 10 monclient: dump: 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.019+0000 7f033959a640 4 set_mon_vals no callback set 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.020+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f0324004ba0 con 0x7f033c13f710 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.020+0000 7f033a59c640 10 monclient: get_auth_request con 0x7f031c06dd00 auth_method 0 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.021+0000 7f0332ffd640 10 client.4625.objecter ms_handle_connect 0x7f031c06dd00 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f03240547d0 con 0x7f033c13f710 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 10 client.4625.objecter ms_dispatch 0x7f033c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0324050110 con 0x7f033c13f710 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.026+0000 7f0332ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:15.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.247+0000 7f0340fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.247+0000 7f0340fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.247+0000 7f0340fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:15.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.247+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f033c1351a0 con 0x7f033c13f710 2024-03-25T18:14:15.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.287+0000 7f0332ffd640 1 -- 172.21.15.84:0/3175846231 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f0324052210 con 0x7f033c13f710 2024-03-25T18:14:15.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.287+0000 7f0332ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.287+0000 7f0332ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.289 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:15.289 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:15.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.289+0000 7f0340fd0640 10 librados: watch_flush enter 2024-03-25T18:14:15.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 10 librados: watch_flush exit 2024-03-25T18:14:15.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 10 client.4625.objecter successfully canceled tick 2024-03-25T18:14:15.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.157:6800/1801259026 conn(0x7f031c06dd00 legacy=0x7f031c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 10 monclient: shutdown 2024-03-25T18:14:15.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.290+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 >> v1:172.21.15.84:6790/0 conn(0x7f033c13f710 legacy=0x7f033c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.291+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 shutdown_connections 2024-03-25T18:14:15.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.291+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 >> 172.21.15.84:0/3175846231 conn(0x7f033c0b7ea0 msgr2=0x7f033c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.291+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 shutdown_connections 2024-03-25T18:14:15.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.291+0000 7f0340fd0640 1 -- 172.21.15.84:0/3175846231 wait complete. 2024-03-25T18:14:15.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:15.291+0000 7f0340fd0640 1 librados: shutdown 2024-03-25T18:14:15.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:22.111 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.227+0000 7ff641954640 10 monclient: get_monmap_and_config 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.227+0000 7ff641954640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.227+0000 7ff641954640 10 monclient: monmap: 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:22.228 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:22.228486+0000 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:22.228486+0000 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:22.229 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 1 Processor -- start 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 1 -- start start 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient: init 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient(hunting): picked mon.noname-c con 0x7ff63c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient(hunting): picked mon.noname-b con 0x7ff63c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient(hunting): picked mon.noname-a con 0x7ff63c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:22.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c142520 con 0x7ff63c1430c0 2024-03-25T18:14:22.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0b28a0 con 0x7ff63c0a98f0 2024-03-25T18:14:22.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0b2a30 con 0x7ff63c13f710 2024-03-25T18:14:22.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.228+0000 7ff641954640 10 monclient(hunting): authenticate will time out at 7506.527696s 2024-03-25T18:14:22.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.232+0000 7ff63affd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff63c0a98f0 0x7ff63c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34658/0 (socket says 172.21.15.84:34658) 2024-03-25T18:14:22.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.232+0000 7ff63affd640 1 -- 172.21.15.84:0/4234130426 learned_addr learned my addr 172.21.15.84:0/4234130426 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363002379 0 0) 0x7ff63c0b28a0 con 0x7ff63c0a98f0 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:14:22.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.247+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff628003520 con 0x7ff63c0a98f0 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633734355 0 0) 0x7ff63c142520 con 0x7ff63c1430c0 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:14:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.249+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c0b28a0 con 0x7ff63c1430c0 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.252+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132626296 0 0) 0x7ff63c0b2a30 con 0x7ff63c13f710 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.252+0000 7ff63a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.252+0000 7ff63a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.252+0000 7ff63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.253+0000 7ff63a7fc640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:14:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.253+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c142520 con 0x7ff63c13f710 2024-03-25T18:14:22.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170558449 0 0) 0x7ff628003520 con 0x7ff63c0a98f0 2024-03-25T18:14:22.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 >> v1:172.21.15.84:6790/0 conn(0x7ff63c13f710 legacy=0x7ff63c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 >> v1:172.21.15.84:6789/0 conn(0x7ff63c1430c0 legacy=0x7ff63c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0b44c0 con 0x7ff63c0a98f0 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff62c001690 con 0x7ff63c0a98f0 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff63a7fc640 10 monclient: dump: 2024-03-25T18:14:22.257 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff641954640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:14:22.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.255+0000 7ff641954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.260+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff62c001870 con 0x7ff63c0a98f0 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff63a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff63a7fc640 1 -- 172.21.15.84:0/4234130426 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff62c001c80 con 0x7ff63c0a98f0 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff641954640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff641954640 4 set_mon_vals no callback set 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff63a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff63a7fc640 10 monclient: dump: 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:22.262 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:22.263 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff641954640 10 monclient: shutdown 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff641954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.261+0000 7ff641954640 1 -- 172.21.15.84:0/4234130426 >> v1:172.21.15.157:6789/0 conn(0x7ff63c0a98f0 legacy=0x7ff63c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.262+0000 7ff641954640 1 -- 172.21.15.84:0/4234130426 shutdown_connections 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.262+0000 7ff641954640 1 -- 172.21.15.84:0/4234130426 >> 172.21.15.84:0/4234130426 conn(0x7ff63c0b7ea0 msgr2=0x7ff63c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.262+0000 7ff641954640 1 -- 172.21.15.84:0/4234130426 shutdown_connections 2024-03-25T18:14:22.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.262+0000 7ff641954640 1 -- 172.21.15.84:0/4234130426 wait complete. 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 10 monclient: monmap: 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:22.265 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 librados: starting msgr at 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 librados: starting objecter 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 Processor -- start 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 -- start start 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 librados: setting wanted keys 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 1 librados: calling monclient init 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.263+0000 7ff641954640 10 monclient: init 2024-03-25T18:14:22.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient(hunting): picked mon.noname-c con 0x7ff63c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient(hunting): picked mon.noname-b con 0x7ff63c13f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient(hunting): picked mon.noname-a con 0x7ff63c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c1d76a0 con 0x7ff63c1430c0 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c1d82b0 con 0x7ff63c13f710 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c1d8440 con 0x7ff63c0a98f0 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.264+0000 7ff641954640 10 monclient(hunting): authenticate will time out at 7506.563305s 2024-03-25T18:14:22.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.270+0000 7ff633fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff63c13f710 0x7ff63c1386d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:34674/0 (socket says 172.21.15.84:34674) 2024-03-25T18:14:22.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.270+0000 7ff633fff640 1 -- 172.21.15.84:0/997046675 learned_addr learned my addr 172.21.15.84:0/997046675 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:22.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.270+0000 7ff638ff9640 10 client.?.objecter ms_handle_connect 0x7ff63c13f710 2024-03-25T18:14:22.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.270+0000 7ff638ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732307273 0 0) 0x7ff63c1d82b0 con 0x7ff63c13f710 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:14:22.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.279+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c004eb0 con 0x7ff63c13f710 2024-03-25T18:14:22.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.284+0000 7ff638ff9640 10 client.?.objecter ms_handle_connect 0x7ff63c1430c0 2024-03-25T18:14:22.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.284+0000 7ff638ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.288+0000 7ff638ff9640 10 client.?.objecter ms_handle_connect 0x7ff63c0a98f0 2024-03-25T18:14:22.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.288+0000 7ff638ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1174260211 0 0) 0x7ff61c004eb0 con 0x7ff63c13f710 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff63c1d82b0 con 0x7ff63c13f710 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff624001a10 con 0x7ff63c13f710 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:22.293 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:22.294 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:22.294 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:22.294 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.291+0000 7ff638ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772055777 0 0) 0x7ff63c1d8440 con 0x7ff63c0a98f0 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:14:22.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.298+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c004eb0 con 0x7ff63c0a98f0 2024-03-25T18:14:22.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 362616051 0 0) 0x7ff63c1d82b0 con 0x7ff63c13f710 2024-03-25T18:14:22.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 >> v1:172.21.15.84:6790/0 conn(0x7ff63c0a98f0 legacy=0x7ff63c1d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 >> v1:172.21.15.84:6789/0 conn(0x7ff63c1430c0 legacy=0x7ff63c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 1 monclient: found mon.b 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.302+0000 7ff638ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff638ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c138de0 con 0x7ff63c13f710 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff638ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff638ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff638ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 10 monclient: _renew_subs 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c137f30 con 0x7ff63c13f710 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 10 monclient: _renew_subs 2024-03-25T18:14:22.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c135340 con 0x7ff63c13f710 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.303+0000 7ff641954640 1 librados: init done 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.304+0000 7ff641954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.304+0000 7ff641954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.304+0000 7ff641954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.304+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c139030 con 0x7ff63c13f710 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff624001cb0 con 0x7ff63c13f710 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7ff624003a60 con 0x7ff63c13f710 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff639ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:22.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.576+0000 7ff638ff9640 10 monclient: dump: 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:22.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.577+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7ff624003c80 con 0x7ff63c13f710 2024-03-25T18:14:22.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.578+0000 7ff63affd640 10 monclient: get_auth_request con 0x7ff61c06dd90 auth_method 0 2024-03-25T18:14:22.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.578+0000 7ff638ff9640 10 client.4639.objecter ms_handle_connect 0x7ff61c06dd90 2024-03-25T18:14:22.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.582+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7ff624054850 con 0x7ff63c13f710 2024-03-25T18:14:22.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.582+0000 7ff638ff9640 10 client.4639.objecter ms_dispatch 0x7ff63c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:22.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.582+0000 7ff638ff9640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:22.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.582+0000 7ff638ff9640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:22.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.582+0000 7ff638ff9640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:14:22.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.583+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff624050360 con 0x7ff63c13f710 2024-03-25T18:14:22.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.583+0000 7ff638ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.583+0000 7ff638ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:22.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.795+0000 7ff641954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:22.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.795+0000 7ff641954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:22.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.795+0000 7ff641954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:22.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.795+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff63c0abde0 con 0x7ff63c13f710 2024-03-25T18:14:22.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.821+0000 7ff638ff9640 1 -- 172.21.15.84:0/997046675 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7ff624052290 con 0x7ff63c13f710 2024-03-25T18:14:22.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.822+0000 7ff638ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:22.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.822+0000 7ff638ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:22.823 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:22.823 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 10 librados: watch_flush enter 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 10 librados: watch_flush exit 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 >> v1:172.21.15.157:6800/1801259026 conn(0x7ff61c06dd90 legacy=0x7ff61c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 10 monclient: shutdown 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.823+0000 7ff641954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 >> v1:172.21.15.157:6789/0 conn(0x7ff63c13f710 legacy=0x7ff63c1386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 shutdown_connections 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 >> 172.21.15.84:0/997046675 conn(0x7ff63c0b7ea0 msgr2=0x7ff63c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 shutdown_connections 2024-03-25T18:14:22.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 -- 172.21.15.84:0/997046675 wait complete. 2024-03-25T18:14:22.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:22.824+0000 7ff641954640 1 librados: shutdown 2024-03-25T18:14:22.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:23.931 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:20.749 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:23.987 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:20.806 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.022 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:20.841 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.029 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:20.813 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.034 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:20.818 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.038 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:20.857 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.044 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:20.863 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.060 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:20.844 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.070 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:20.889 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.642 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.751+0000 7f55bf40f640 10 monclient: get_monmap_and_config 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.751+0000 7f55bf40f640 10 monclient: build_initial_monmap 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.751+0000 7f55bf40f640 10 monclient: monmap: 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:29.753091+0000 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:29.753091+0000 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:29.753 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:29.754 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:29.754 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.752+0000 7f55bf40f640 1 Processor -- start 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.752+0000 7f55bf40f640 1 -- start start 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.752+0000 7f55bf40f640 10 monclient: init 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-c con 0x7f55b80b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-a con 0x7f55b80b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-b con 0x7f55b814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b80af940 con 0x7f55b80b1730 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b80b0a70 con 0x7f55b80b4d60 2024-03-25T18:14:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b80b0d50 con 0x7f55b814be90 2024-03-25T18:14:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.753+0000 7f55bf40f640 10 monclient(hunting): authenticate will time out at 7514.052197s 2024-03-25T18:14:29.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.756+0000 7f55bd9ac640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f55b814be90 0x7f55b814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57926/0 (socket says 172.21.15.84:57926) 2024-03-25T18:14:29.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.756+0000 7f55bd9ac640 1 -- 172.21.15.84:0/55085831 learned_addr learned my addr 172.21.15.84:0/55085831 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.766+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530262784 0 0) 0x7f55b80af940 con 0x7f55b80b1730 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a4003520 con 0x7f55b80b1730 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858417565 0 0) 0x7f55b80b0d50 con 0x7f55b814be90 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:14:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.767+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b80af940 con 0x7f55b814be90 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.768+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119373243 0 0) 0x7f55b80b0a70 con 0x7f55b80b4d60 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.769+0000 7f55affff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.769+0000 7f55affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.769+0000 7f55affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.769+0000 7f55affff640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:14:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.769+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b80b0d50 con 0x7f55b80b4d60 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165273007 0 0) 0x7f55a4003520 con 0x7f55b80b1730 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 >> v1:172.21.15.84:6790/0 conn(0x7f55b814be90 legacy=0x7f55b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 >> v1:172.21.15.157:6789/0 conn(0x7f55b80b4d60 legacy=0x7f55b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 monclient: found mon.noname-a 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80b1030 con 0x7f55b80b1730 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: _finish_auth 0 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:29.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f55a8001690 con 0x7f55b80b1730 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55bf40f640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55bf40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55affff640 10 monclient: dump: 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.777+0000 7f55bf40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55a80017f0 con 0x7f55b80b1730 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 1 -- 172.21.15.84:0/55085831 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f55a8001c00 con 0x7f55b80b1730 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:29.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55affff640 10 monclient: dump: 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:29.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 4 set_mon_vals no callback set 2024-03-25T18:14:29.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 10 monclient: shutdown 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 1 -- 172.21.15.84:0/55085831 >> v1:172.21.15.84:6789/0 conn(0x7f55b80b1730 legacy=0x7f55b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 1 -- 172.21.15.84:0/55085831 shutdown_connections 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.785+0000 7f55bf40f640 1 -- 172.21.15.84:0/55085831 >> 172.21.15.84:0/55085831 conn(0x7f55b80617a0 msgr2=0x7f55b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.786+0000 7f55bf40f640 1 -- 172.21.15.84:0/55085831 shutdown_connections 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.786+0000 7f55bf40f640 1 -- 172.21.15.84:0/55085831 wait complete. 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.786+0000 7f55bf40f640 10 monclient: build_initial_monmap 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.786+0000 7f55bf40f640 10 monclient: monmap: 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:29.790 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 librados: starting msgr at 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 librados: starting objecter 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 Processor -- start 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 -- start start 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 librados: setting wanted keys 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 1 librados: calling monclient init 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.787+0000 7f55bf40f640 10 monclient: init 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-b con 0x7f55b80b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:29.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-a con 0x7f55b80b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient(hunting): picked mon.noname-c con 0x7f55b814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8138580 con 0x7f55b80b4d60 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8138860 con 0x7f55b80b1730 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8137b90 con 0x7f55b814be90 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:29.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.788+0000 7f55bf40f640 10 monclient(hunting): authenticate will time out at 7514.087203s 2024-03-25T18:14:29.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.795+0000 7f55bd9ac640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f55b814be90 0x7f55b813a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57956/0 (socket says 172.21.15.84:57956) 2024-03-25T18:14:29.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.795+0000 7f55bd9ac640 1 -- 172.21.15.84:0/650922780 learned_addr learned my addr 172.21.15.84:0/650922780 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:29.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.795+0000 7f55adffb640 10 client.?.objecter ms_handle_connect 0x7f55b814be90 2024-03-25T18:14:29.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.795+0000 7f55adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:29.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.803+0000 7f55adffb640 10 client.?.objecter ms_handle_connect 0x7f55b80b4d60 2024-03-25T18:14:29.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.803+0000 7f55adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:29.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.810+0000 7f55adffb640 10 client.?.objecter ms_handle_connect 0x7f55b80b1730 2024-03-25T18:14:29.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.810+0000 7f55adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972720568 0 0) 0x7f55b8137b90 con 0x7f55b814be90 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:14:29.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.812+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c004eb0 con 0x7f55b814be90 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716856671 0 0) 0x7f55b8138580 con 0x7f55b80b4d60 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:14:29.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.818+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b8137b90 con 0x7f55b80b4d60 2024-03-25T18:14:29.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337883196 0 0) 0x7f559c004eb0 con 0x7f55b814be90 2024-03-25T18:14:29.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b8138580 con 0x7f55b814be90 2024-03-25T18:14:29.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f55b0001c70 con 0x7f55b814be90 2024-03-25T18:14:29.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 10 monclient(hunting): dump: 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.822+0000 7f55adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:29.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193768165 0 0) 0x7f55b8138860 con 0x7f55b80b1730 2024-03-25T18:14:29.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:29.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:29.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:29.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:14:29.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.829+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c004eb0 con 0x7f55b80b1730 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014756440 0 0) 0x7f55b8137b90 con 0x7f55b80b4d60 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b8138860 con 0x7f55b80b4d60 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f55a8001b00 con 0x7f55b80b4d60 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.834+0000 7f55adffb640 10 monclient(hunting): dump: 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:29.836 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:29.837 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:29.837 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:29.837 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:29.837 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 792590759 0 0) 0x7f55b8138580 con 0x7f55b814be90 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 >> v1:172.21.15.157:6789/0 conn(0x7f55b80b1730 legacy=0x7f55b8137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 >> v1:172.21.15.84:6789/0 conn(0x7f55b80b4d60 legacy=0x7f55b8131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 1 monclient: found mon.c 2024-03-25T18:14:29.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8131890 con 0x7f55b814be90 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 10 monclient: _renew_subs 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b8131b40 con 0x7f55b814be90 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 10 monclient: _renew_subs 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.840+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b813b080 con 0x7f55b814be90 2024-03-25T18:14:29.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.841+0000 7f55bf40f640 1 librados: init done 2024-03-25T18:14:29.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.842+0000 7f55bf40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:29.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.842+0000 7f55bf40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:29.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.842+0000 7f55bf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:29.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:29.842+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b8132980 con 0x7f55b814be90 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55b0004b60 con 0x7f55b814be90 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f55b0004cd0 con 0x7f55b814be90 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55adffb640 10 monclient: dump: 2024-03-25T18:14:30.179 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.177+0000 7f55af7fe640 4 set_mon_vals no callback set 2024-03-25T18:14:30.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.178+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f55b001aca0 con 0x7f55b814be90 2024-03-25T18:14:30.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.179+0000 7f55bd1ab640 10 monclient: get_auth_request con 0x7f559c06df40 auth_method 0 2024-03-25T18:14:30.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.179+0000 7f55adffb640 10 client.4637.objecter ms_handle_connect 0x7f559c06df40 2024-03-25T18:14:30.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f55b00548e0 con 0x7f55b814be90 2024-03-25T18:14:30.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 10 client.4637.objecter ms_dispatch 0x7f55b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:30.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:30.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:30.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T18:14:30.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55b0050220 con 0x7f55b814be90 2024-03-25T18:14:30.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.184+0000 7f55adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:30.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.410+0000 7f55bf40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.410+0000 7f55bf40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.410+0000 7f55bf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:14:30.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.410+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55b80627b0 con 0x7f55b814be90 2024-03-25T18:14:30.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.434+0000 7f55adffb640 1 -- 172.21.15.84:0/650922780 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f55b0052320 con 0x7f55b814be90 2024-03-25T18:14:30.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.434+0000 7f55adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.434+0000 7f55adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:30.436 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:30.436 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 10 librados: watch_flush enter 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 10 librados: watch_flush exit 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 10 client.4637.objecter successfully canceled tick 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 >> v1:172.21.15.157:6800/1801259026 conn(0x7f559c06df40 legacy=0x7f559c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 10 monclient: shutdown 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 >> v1:172.21.15.84:6790/0 conn(0x7f55b814be90 legacy=0x7f55b813a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 shutdown_connections 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 >> 172.21.15.84:0/650922780 conn(0x7f55b80617a0 msgr2=0x7f55b80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 shutdown_connections 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 -- 172.21.15.84:0/650922780 wait complete. 2024-03-25T18:14:30.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:30.437+0000 7f55bf40f640 1 librados: shutdown 2024-03-25T18:14:30.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:37.259 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:37.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.373+0000 7f0ee7e9a640 10 monclient: get_monmap_and_config 2024-03-25T18:14:37.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.373+0000 7f0ee7e9a640 10 monclient: build_initial_monmap 2024-03-25T18:14:37.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.373+0000 7f0ee7e9a640 10 monclient: monmap: 2024-03-25T18:14:37.375 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:37.375306+0000 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:37.375306+0000 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:37.376 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.374+0000 7f0ee7e9a640 1 Processor -- start 2024-03-25T18:14:37.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 1 -- start start 2024-03-25T18:14:37.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient: init 2024-03-25T18:14:37.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee00b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee00b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00af940 con 0x7f0ee00b4d60 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00b0a70 con 0x7f0ee014be90 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00b0d50 con 0x7f0ee00b1730 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:37.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.375+0000 7f0ee7e9a640 10 monclient(hunting): authenticate will time out at 7521.674731s 2024-03-25T18:14:37.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.380+0000 7f0ee6437640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0ee014be90 0x7f0ee014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59106/0 (socket says 172.21.15.84:59106) 2024-03-25T18:14:37.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.380+0000 7f0ee6437640 1 -- 172.21.15.84:0/1271338078 learned_addr learned my addr 172.21.15.84:0/1271338078 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:37.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638535626 0 0) 0x7f0ee00b0a70 con 0x7f0ee014be90 2024-03-25T18:14:37.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:14:37.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.392+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ecc003520 con 0x7f0ee014be90 2024-03-25T18:14:37.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204914872 0 0) 0x7f0ee00b0d50 con 0x7f0ee00b1730 2024-03-25T18:14:37.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:14:37.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.401+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee00b0a70 con 0x7f0ee00b1730 2024-03-25T18:14:37.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194545976 0 0) 0x7f0ecc003520 con 0x7f0ee014be90 2024-03-25T18:14:37.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 >> v1:172.21.15.84:6790/0 conn(0x7f0ee00b1730 legacy=0x7f0ee00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 >> v1:172.21.15.84:6789/0 conn(0x7f0ee00b4d60 legacy=0x7f0ee013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: _finish_hunting 0 2024-03-25T18:14:37.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 monclient: found mon.noname-c 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00b1030 con 0x7f0ee014be90 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: _finish_auth 0 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: _check_auth_tickets 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0ed8001a10 con 0x7f0ee014be90 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee7e9a640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee7e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee4c34640 10 monclient: dump: 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:37.406 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:37.407 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.402+0000 7f0ee7e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ed8001b70 con 0x7f0ee014be90 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 1 -- 172.21.15.84:0/1271338078 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0ed8001f80 con 0x7f0ee014be90 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee4c34640 10 monclient: dump: 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.419 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee7e9a640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:37.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.417+0000 7f0ee7e9a640 4 set_mon_vals no callback set 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 10 monclient: shutdown 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1271338078 >> v1:172.21.15.157:6789/0 conn(0x7f0ee014be90 legacy=0x7f0ee014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1271338078 shutdown_connections 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1271338078 >> 172.21.15.84:0/1271338078 conn(0x7f0ee00617a0 msgr2=0x7f0ee0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1271338078 shutdown_connections 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.418+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1271338078 wait complete. 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.419+0000 7f0ee7e9a640 10 monclient: build_initial_monmap 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.419+0000 7f0ee7e9a640 10 monclient: monmap: 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:37.423 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 librados: starting msgr at 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 librados: starting objecter 2024-03-25T18:14:37.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 Processor -- start 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 -- start start 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 librados: setting wanted keys 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 1 librados: calling monclient init 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.420+0000 7f0ee7e9a640 10 monclient: init 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee00b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee00b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee0135ee0 con 0x7f0ee00b4d60 2024-03-25T18:14:37.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee0135b00 con 0x7f0ee014be90 2024-03-25T18:14:37.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee0135c90 con 0x7f0ee00b1730 2024-03-25T18:14:37.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:37.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.421+0000 7f0ee7e9a640 10 monclient(hunting): authenticate will time out at 7521.720242s 2024-03-25T18:14:37.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.424+0000 7f0ee6437640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0ee014be90 0x7f0ee01d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:59146/0 (socket says 172.21.15.84:59146) 2024-03-25T18:14:37.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.424+0000 7f0ee6437640 1 -- 172.21.15.84:0/1576995483 learned_addr learned my addr 172.21.15.84:0/1576995483 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:37.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.427+0000 7f0ed67fc640 10 client.?.objecter ms_handle_connect 0x7f0ee014be90 2024-03-25T18:14:37.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.427+0000 7f0ed67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:37.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.433+0000 7f0ed67fc640 10 client.?.objecter ms_handle_connect 0x7f0ee00b1730 2024-03-25T18:14:37.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.433+0000 7f0ed67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:37.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422255964 0 0) 0x7f0ee0135b00 con 0x7f0ee014be90 2024-03-25T18:14:37.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:14:37.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.435+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec4004eb0 con 0x7f0ee014be90 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 870840792 0 0) 0x7f0ec4004eb0 con 0x7f0ee014be90 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ee0135b00 con 0x7f0ee014be90 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0ed8001ec0 con 0x7f0ee014be90 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 10 monclient(hunting): dump: 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.446 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.444+0000 7f0ed67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4131322252 0 0) 0x7f0ee0135b00 con 0x7f0ee014be90 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 >> v1:172.21.15.84:6790/0 conn(0x7f0ee00b1730 legacy=0x7f0ee012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 >> v1:172.21.15.84:6789/0 conn(0x7f0ee00b4d60 legacy=0x7f0ee01d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 1 monclient: found mon.b 2024-03-25T18:14:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee0136960 con 0x7f0ee014be90 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 10 monclient: _renew_subs 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee0130470 con 0x7f0ee014be90 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 10 monclient: _renew_subs 2024-03-25T18:14:37.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee013e870 con 0x7f0ee014be90 2024-03-25T18:14:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.448+0000 7f0ee7e9a640 1 librados: init done 2024-03-25T18:14:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.449+0000 7f0ee7e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.449+0000 7f0ee7e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.449+0000 7f0ee7e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.449+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee0130810 con 0x7f0ee014be90 2024-03-25T18:14:37.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ed8004e10 con 0x7f0ee014be90 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f0ed8002020 con 0x7f0ee014be90 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed67fc640 10 monclient: dump: 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:37.475 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.469+0000 7f0ed7fff640 4 set_mon_vals no callback set 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.470+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f0ed801adf0 con 0x7f0ee014be90 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.470+0000 7f0ee5c36640 10 monclient: get_auth_request con 0x7f0ec406db90 auth_method 0 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.471+0000 7f0ed67fc640 10 client.4651.objecter ms_handle_connect 0x7f0ec406db90 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.472+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f0ed80549d0 con 0x7f0ee014be90 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.472+0000 7f0ed67fc640 10 client.4651.objecter ms_dispatch 0x7f0ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.472+0000 7f0ed67fc640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.472+0000 7f0ed67fc640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:37.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.472+0000 7f0ed67fc640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:14:37.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.756+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ed8050310 con 0x7f0ee014be90 2024-03-25T18:14:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.756+0000 7f0ed67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.756+0000 7f0ed67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:37.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.978+0000 7f0ee7e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:37.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.978+0000 7f0ee7e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:37.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.978+0000 7f0ee7e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:37.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:37.978+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ee0130600 con 0x7f0ee014be90 2024-03-25T18:14:38.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.007+0000 7f0ed67fc640 1 -- 172.21.15.84:0/1576995483 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f0ed8052410 con 0x7f0ee014be90 2024-03-25T18:14:38.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.007+0000 7f0ed67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.007+0000 7f0ed67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:38.009 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:38.009 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 10 librados: watch_flush enter 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 10 librados: watch_flush exit 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 >> v1:172.21.15.157:6800/1801259026 conn(0x7f0ec406db90 legacy=0x7f0ec4070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 10 monclient: shutdown 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 >> v1:172.21.15.157:6789/0 conn(0x7f0ee014be90 legacy=0x7f0ee01d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 shutdown_connections 2024-03-25T18:14:38.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 >> 172.21.15.84:0/1576995483 conn(0x7f0ee00617a0 msgr2=0x7f0ee00b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:38.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 shutdown_connections 2024-03-25T18:14:38.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 -- 172.21.15.84:0/1576995483 wait complete. 2024-03-25T18:14:38.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:38.009+0000 7f0ee7e9a640 1 librados: shutdown 2024-03-25T18:14:38.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:43.997 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:40.816 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.004 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:40.823 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.045 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:40.864 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.052 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:40.871 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.065 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:40.884 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.067 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:40.851 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.075 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:40.859 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.104 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:09:40.924 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.110 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:40.894 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:44.829 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:44.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.938+0000 7fc5ee257640 10 monclient: get_monmap_and_config 2024-03-25T18:14:44.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.938+0000 7fc5ee257640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.938+0000 7fc5ee257640 10 monclient: monmap: 2024-03-25T18:14:44.940 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:44.940 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:44.940306+0000 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:44.940306+0000 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:44.941 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.939+0000 7fc5ee257640 1 Processor -- start 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.939+0000 7fc5ee257640 1 -- start start 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.939+0000 7fc5ee257640 10 monclient: init 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:44.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e80a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e813f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e81430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e8142520 con 0x7fc5e81430c0 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e80b28a0 con 0x7fc5e80a98f0 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e80b2a30 con 0x7fc5e813f710 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.940+0000 7fc5ee257640 10 monclient(hunting): authenticate will time out at 7529.239688s 2024-03-25T18:14:44.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.950+0000 7fc5e77fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc5e80a98f0 0x7fc5e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50548/0 (socket says 172.21.15.84:50548) 2024-03-25T18:14:44.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.950+0000 7fc5e77fe640 1 -- 172.21.15.84:0/47640976 learned_addr learned my addr 172.21.15.84:0/47640976 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.962+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572557201 0 0) 0x7fc5e80b28a0 con 0x7fc5e80a98f0 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.963+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.963+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.963+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.963+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:14:44.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.963+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d0003520 con 0x7fc5e80a98f0 2024-03-25T18:14:44.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206570175 0 0) 0x7fc5e80b2a30 con 0x7fc5e813f710 2024-03-25T18:14:44.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:14:44.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.977+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e80b28a0 con 0x7fc5e813f710 2024-03-25T18:14:44.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1925057871 0 0) 0x7fc5d0003520 con 0x7fc5e80a98f0 2024-03-25T18:14:44.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 >> v1:172.21.15.84:6790/0 conn(0x7fc5e813f710 legacy=0x7fc5e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 >> v1:172.21.15.84:6789/0 conn(0x7fc5e81430c0 legacy=0x7fc5e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80b44c0 con 0x7fc5e80a98f0 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc5d8001690 con 0x7fc5e80a98f0 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5e67fc640 10 monclient: dump: 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:44.982 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5ee257640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:14:44.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.980+0000 7fc5ee257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:44.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5d80017f0 con 0x7fc5e80a98f0 2024-03-25T18:14:44.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 1 -- 172.21.15.84:0/47640976 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc5d8001c00 con 0x7fc5e80a98f0 2024-03-25T18:14:44.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5e67fc640 10 monclient: dump: 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5ee257640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:44.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.996+0000 7fc5ee257640 4 set_mon_vals no callback set 2024-03-25T18:14:44.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 10 monclient: shutdown 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 1 -- 172.21.15.84:0/47640976 >> v1:172.21.15.157:6789/0 conn(0x7fc5e80a98f0 legacy=0x7fc5e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 1 -- 172.21.15.84:0/47640976 shutdown_connections 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 1 -- 172.21.15.84:0/47640976 >> 172.21.15.84:0/47640976 conn(0x7fc5e80b7ea0 msgr2=0x7fc5e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 1 -- 172.21.15.84:0/47640976 shutdown_connections 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.997+0000 7fc5ee257640 1 -- 172.21.15.84:0/47640976 wait complete. 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.998+0000 7fc5ee257640 10 monclient: build_initial_monmap 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.998+0000 7fc5ee257640 10 monclient: monmap: 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:45.000 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.998+0000 7fc5ee257640 1 librados: starting msgr at 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.998+0000 7fc5ee257640 1 librados: starting objecter 2024-03-25T18:14:45.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.998+0000 7fc5ee257640 1 Processor -- start 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 1 -- start start 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 1 librados: setting wanted keys 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 1 librados: calling monclient init 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 10 monclient: init 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e80a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:44.999+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e813f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e81430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:45.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e8135730 con 0x7fc5e813f710 2024-03-25T18:14:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e81358c0 con 0x7fc5e80a98f0 2024-03-25T18:14:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e8135a50 con 0x7fc5e81430c0 2024-03-25T18:14:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.000+0000 7fc5ee257640 10 monclient(hunting): authenticate will time out at 7529.299144s 2024-03-25T18:14:45.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.021+0000 7fc5e77fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc5e80a98f0 0x7fc5e81dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:50616/0 (socket says 172.21.15.84:50616) 2024-03-25T18:14:45.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.021+0000 7fc5e77fe640 1 -- 172.21.15.84:0/423691350 learned_addr learned my addr 172.21.15.84:0/423691350 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.023+0000 7fc5cffff640 10 client.?.objecter ms_handle_connect 0x7fc5e80a98f0 2024-03-25T18:14:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.023+0000 7fc5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.027+0000 7fc5cffff640 10 client.?.objecter ms_handle_connect 0x7fc5e813f710 2024-03-25T18:14:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.027+0000 7fc5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264752967 0 0) 0x7fc5e81358c0 con 0x7fc5e80a98f0 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:14:45.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.043+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4004eb0 con 0x7fc5e80a98f0 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532356181 0 0) 0x7fc5e8135730 con 0x7fc5e813f710 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:14:45.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.044+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e81358c0 con 0x7fc5e813f710 2024-03-25T18:14:45.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.055+0000 7fc5cffff640 10 client.?.objecter ms_handle_connect 0x7fc5e81430c0 2024-03-25T18:14:45.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.055+0000 7fc5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 124818428 0 0) 0x7fc5e81358c0 con 0x7fc5e813f710 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5e8135730 con 0x7fc5e813f710 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc5dc001930 con 0x7fc5e813f710 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 10 monclient(hunting): dump: 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:45.059 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:45.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.057+0000 7fc5cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.069+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 267888829 0 0) 0x7fc5d4004eb0 con 0x7fc5e80a98f0 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.070+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5e81358c0 con 0x7fc5e80a98f0 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.070+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc5d8001c40 con 0x7fc5e80a98f0 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.070+0000 7fc5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.070+0000 7fc5cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.070+0000 7fc5cffff640 10 monclient(hunting): dump: 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:45.071 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:45.072 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2145082181 0 0) 0x7fc5e8135730 con 0x7fc5e813f710 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 >> v1:172.21.15.84:6790/0 conn(0x7fc5e81430c0 legacy=0x7fc5e81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 >> v1:172.21.15.157:6789/0 conn(0x7fc5e80a98f0 legacy=0x7fc5e81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 1 monclient: found mon.a 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e81d5000 con 0x7fc5e813f710 2024-03-25T18:14:45.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 10 monclient: _renew_subs 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e81d52b0 con 0x7fc5e813f710 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 10 monclient: _renew_subs 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e813c900 con 0x7fc5e813f710 2024-03-25T18:14:45.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.081+0000 7fc5ee257640 1 librados: init done 2024-03-25T18:14:45.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.082+0000 7fc5ee257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.082+0000 7fc5ee257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.082+0000 7fc5ee257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:14:45.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.082+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e81371c0 con 0x7fc5e813f710 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5dc003610 con 0x7fc5e813f710 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc5dc0039e0 con 0x7fc5e813f710 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5cffff640 10 monclient: dump: 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.431 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.430+0000 7fc5e5ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.431+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fc5dc003c00 con 0x7fc5e813f710 2024-03-25T18:14:45.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.431+0000 7fc5e77fe640 10 monclient: get_auth_request con 0x7fc5d406de90 auth_method 0 2024-03-25T18:14:45.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.432+0000 7fc5cffff640 10 client.4659.objecter ms_handle_connect 0x7fc5d406de90 2024-03-25T18:14:45.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7fc5dc0535c0 con 0x7fc5e813f710 2024-03-25T18:14:45.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 10 client.4659.objecter ms_dispatch 0x7fc5e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5dc04f0d0 con 0x7fc5e813f710 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.439+0000 7fc5cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:45.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.657+0000 7fc5ee257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.658+0000 7fc5ee257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.658+0000 7fc5ee257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:14:45.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.658+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5e80627b0 con 0x7fc5e813f710 2024-03-25T18:14:45.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.700+0000 7fc5cffff640 1 -- 172.21.15.84:0/423691350 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7fc5dc051000 con 0x7fc5e813f710 2024-03-25T18:14:45.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.700+0000 7fc5cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.700+0000 7fc5cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:45.701 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:45.701 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 10 librados: watch_flush enter 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 10 librados: watch_flush exit 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 >> v1:172.21.15.157:6800/1801259026 conn(0x7fc5d406de90 legacy=0x7fc5d4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 10 monclient: shutdown 2024-03-25T18:14:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 >> v1:172.21.15.84:6789/0 conn(0x7fc5e813f710 legacy=0x7fc5e81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 shutdown_connections 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 >> 172.21.15.84:0/423691350 conn(0x7fc5e80b7ea0 msgr2=0x7fc5e814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 shutdown_connections 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 -- 172.21.15.84:0/423691350 wait complete. 2024-03-25T18:14:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:45.703+0000 7fc5ee257640 1 librados: shutdown 2024-03-25T18:14:45.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:52.523 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.634+0000 7f50139f6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.634+0000 7f50139f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.634+0000 7f50139f6640 10 monclient: monmap: 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:14:52.635120+0000 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:14:52.635120+0000 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:52.635 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:52.636 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:14:52.636 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:14:52.636 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.635+0000 7f50139f6640 1 Processor -- start 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.635+0000 7f50139f6640 1 -- start start 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.635+0000 7f50139f6640 10 monclient: init 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-c con 0x7f500c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-b con 0x7f500c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:52.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-a con 0x7f500c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:52.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c142520 con 0x7f500c1430c0 2024-03-25T18:14:52.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0b28a0 con 0x7f500c0a98f0 2024-03-25T18:14:52.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0b2a30 con 0x7f500c13f710 2024-03-25T18:14:52.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:52.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.636+0000 7f50139f6640 10 monclient(hunting): authenticate will time out at 7536.934417s 2024-03-25T18:14:52.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.649+0000 7f5011792640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f500c0a98f0 0x7f500c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:48678/0 (socket says 172.21.15.84:48678) 2024-03-25T18:14:52.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.649+0000 7f5011792640 1 -- 172.21.15.84:0/3450413484 learned_addr learned my addr 172.21.15.84:0/3450413484 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067460305 0 0) 0x7f500c0b28a0 con 0x7f500c0a98f0 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:14:52.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.658+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003520 con 0x7f500c0a98f0 2024-03-25T18:14:52.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869908093 0 0) 0x7f500c0b2a30 con 0x7f500c13f710 2024-03-25T18:14:52.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:14:52.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.662+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f500c0b28a0 con 0x7f500c13f710 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928369664 0 0) 0x7f500c142520 con 0x7f500c1430c0 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:14:52.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.673+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f500c0b2a30 con 0x7f500c1430c0 2024-03-25T18:14:52.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143799638 0 0) 0x7f4ff8003520 con 0x7f500c0a98f0 2024-03-25T18:14:52.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 >> v1:172.21.15.84:6790/0 conn(0x7f500c13f710 legacy=0x7f500c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:52.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 >> v1:172.21.15.84:6789/0 conn(0x7f500c1430c0 legacy=0x7f500c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:52.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:52.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 monclient: found mon.noname-c 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0b44c0 con 0x7f500c0a98f0 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f50139f6640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5008001690 con 0x7f500c0a98f0 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.675+0000 7f5003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.676+0000 7f5003fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.676+0000 7f5003fff640 10 monclient: dump: 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.677 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:52.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.676+0000 7f50139f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:52.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.679+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50080017f0 con 0x7f500c0a98f0 2024-03-25T18:14:52.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f5003fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:52.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f5003fff640 1 -- 172.21.15.84:0/3450413484 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5008001c00 con 0x7f500c0a98f0 2024-03-25T18:14:52.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:52.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 4 set_mon_vals no callback set 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f5003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f5003fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f5003fff640 10 monclient: dump: 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:52.681 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 10 monclient: shutdown 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 1 -- 172.21.15.84:0/3450413484 >> v1:172.21.15.157:6789/0 conn(0x7f500c0a98f0 legacy=0x7f500c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 1 -- 172.21.15.84:0/3450413484 shutdown_connections 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.680+0000 7f50139f6640 1 -- 172.21.15.84:0/3450413484 >> 172.21.15.84:0/3450413484 conn(0x7f500c0b7ea0 msgr2=0x7f500c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.681+0000 7f50139f6640 1 -- 172.21.15.84:0/3450413484 shutdown_connections 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.681+0000 7f50139f6640 1 -- 172.21.15.84:0/3450413484 wait complete. 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.681+0000 7f50139f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:52.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.681+0000 7f50139f6640 10 monclient: monmap: 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 librados: starting msgr at 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 librados: starting objecter 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 Processor -- start 2024-03-25T18:14:52.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 -- start start 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 librados: setting wanted keys 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 1 librados: calling monclient init 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.682+0000 7f50139f6640 10 monclient: init 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-a con 0x7f500c0a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-c con 0x7f500c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient(hunting): picked mon.noname-b con 0x7f500c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c139870 con 0x7f500c0a98f0 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c1d51f0 con 0x7f500c1430c0 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c138830 con 0x7f500c13f710 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:52.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.683+0000 7f50139f6640 10 monclient(hunting): authenticate will time out at 7536.981058s 2024-03-25T18:14:52.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.692+0000 7f5010f91640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f500c13f710 0x7f500c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53468/0 (socket says 172.21.15.84:53468) 2024-03-25T18:14:52.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.692+0000 7f5010f91640 1 -- 172.21.15.84:0/152486984 learned_addr learned my addr 172.21.15.84:0/152486984 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:14:52.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.695+0000 7f5001ffb640 10 client.?.objecter ms_handle_connect 0x7f500c13f710 2024-03-25T18:14:52.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.695+0000 7f5001ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:52.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.697+0000 7f5001ffb640 10 client.?.objecter ms_handle_connect 0x7f500c1430c0 2024-03-25T18:14:52.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.697+0000 7f5001ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:52.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879328164 0 0) 0x7f500c1d51f0 con 0x7f500c1430c0 2024-03-25T18:14:52.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:14:52.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.708+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0004eb0 con 0x7f500c1430c0 2024-03-25T18:14:52.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 client.?.objecter ms_handle_connect 0x7f500c0a98f0 2024-03-25T18:14:52.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:52.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935526277 0 0) 0x7f500c138830 con 0x7f500c13f710 2024-03-25T18:14:52.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:14:52.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.714+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f500c1d51f0 con 0x7f500c13f710 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052769113 0 0) 0x7f4ff0004eb0 con 0x7f500c1430c0 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f500c138830 con 0x7f500c1430c0 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5004001bd0 con 0x7f500c1430c0 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:52.721 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:52.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.721+0000 7f5001ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:52.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622254119 0 0) 0x7f500c139870 con 0x7f500c0a98f0 2024-03-25T18:14:52.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:52.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:52.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:52.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:14:52.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.729+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0004eb0 con 0x7f500c0a98f0 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1364961667 0 0) 0x7f500c138830 con 0x7f500c1430c0 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 >> v1:172.21.15.84:6790/0 conn(0x7f500c13f710 legacy=0x7f500c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 >> v1:172.21.15.84:6789/0 conn(0x7f500c0a98f0 legacy=0x7f500c138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 1 monclient: found mon.b 2024-03-25T18:14:52.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c1d77a0 con 0x7f500c1430c0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f5001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f50139f6640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f50139f6640 10 monclient: _renew_subs 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f50139f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.732+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c1d83d0 con 0x7f500c1430c0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.733+0000 7f50139f6640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.733+0000 7f50139f6640 10 monclient: _renew_subs 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.733+0000 7f50139f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.733+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c138c80 con 0x7f500c1430c0 2024-03-25T18:14:52.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.733+0000 7f50139f6640 1 librados: init done 2024-03-25T18:14:52.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.734+0000 7f50139f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:52.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.734+0000 7f50139f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:52.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.734+0000 7f50139f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:52.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:52.734+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c0627b0 con 0x7f500c1430c0 2024-03-25T18:14:53.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5004002060 con 0x7f500c1430c0 2024-03-25T18:14:53.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f5004003b30 con 0x7f500c1430c0 2024-03-25T18:14:53.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f5001ffb640 10 monclient: dump: 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:14:53.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.036+0000 7f50037fe640 4 set_mon_vals no callback set 2024-03-25T18:14:53.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.037+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f5004003d50 con 0x7f500c1430c0 2024-03-25T18:14:53.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.038+0000 7f5011792640 10 monclient: get_auth_request con 0x7f4ff006ddb0 auth_method 0 2024-03-25T18:14:53.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.038+0000 7f5001ffb640 10 client.4663.objecter ms_handle_connect 0x7f4ff006ddb0 2024-03-25T18:14:53.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.039+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f50040538d0 con 0x7f500c1430c0 2024-03-25T18:14:53.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.039+0000 7f5001ffb640 10 client.4663.objecter ms_dispatch 0x7f500c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:53.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.039+0000 7f5001ffb640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:53.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.039+0000 7f5001ffb640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:53.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.040+0000 7f5001ffb640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:14:53.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.045+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f500404f3e0 con 0x7f500c1430c0 2024-03-25T18:14:53.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.045+0000 7f5001ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.045+0000 7f5001ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:53.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.306+0000 7f50139f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.306+0000 7f50139f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.306+0000 7f50139f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:14:53.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.306+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f500c1d7930 con 0x7f500c1430c0 2024-03-25T18:14:53.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.334+0000 7f5001ffb640 1 -- 172.21.15.84:0/152486984 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f5004051310 con 0x7f500c1430c0 2024-03-25T18:14:53.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.334+0000 7f5001ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.334+0000 7f5001ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:53.335 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:14:53.335 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.336+0000 7f50139f6640 10 librados: watch_flush enter 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 10 librados: watch_flush exit 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 >> v1:172.21.15.157:6800/1801259026 conn(0x7f4ff006ddb0 legacy=0x7f4ff0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 10 monclient: shutdown 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 >> v1:172.21.15.157:6789/0 conn(0x7f500c1430c0 legacy=0x7f500c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 shutdown_connections 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 >> 172.21.15.84:0/152486984 conn(0x7f500c0b7ea0 msgr2=0x7f500c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:53.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 shutdown_connections 2024-03-25T18:14:53.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 -- 172.21.15.84:0/152486984 wait complete. 2024-03-25T18:14:53.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:14:53.337+0000 7f50139f6640 1 librados: shutdown 2024-03-25T18:14:53.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:00.155 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:00.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.269+0000 7f426bc7b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.269+0000 7f426bc7b640 10 monclient: build_initial_monmap 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.269+0000 7f426bc7b640 10 monclient: monmap: 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:00.269704+0000 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:00.269704+0000 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:00.270 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.269+0000 7f426bc7b640 1 Processor -- start 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 1 -- start start 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient: init 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-b con 0x7f42640b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-c con 0x7f42640b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-a con 0x7f426414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42640af940 con 0x7f426414be90 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42640b0a70 con 0x7f42640b1730 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.270+0000 7f426bc7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42640b0d50 con 0x7f42640b4d60 2024-03-25T18:15:00.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.271+0000 7f426bc7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:00.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.271+0000 7f426bc7b640 10 monclient(hunting): authenticate will time out at 7544.568901s 2024-03-25T18:15:00.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.279+0000 7f4269a17640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f42640b4d60 0x7f426413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51520/0 (socket says 172.21.15.84:51520) 2024-03-25T18:15:00.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.279+0000 7f4269a17640 1 -- 172.21.15.84:0/3682200961 learned_addr learned my addr 172.21.15.84:0/3682200961 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307617625 0 0) 0x7f42640b0d50 con 0x7f42640b4d60 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:15:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.302+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4250003520 con 0x7f42640b4d60 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764195225 0 0) 0x7f42640b0a70 con 0x7f42640b1730 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:15:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.304+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42640b0d50 con 0x7f42640b1730 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181056425 0 0) 0x7f42640af940 con 0x7f426414be90 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:15:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.309+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42640b0a70 con 0x7f426414be90 2024-03-25T18:15:00.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1617288261 0 0) 0x7f4250003520 con 0x7f42640b4d60 2024-03-25T18:15:00.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 >> v1:172.21.15.157:6789/0 conn(0x7f42640b1730 legacy=0x7f42640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 >> v1:172.21.15.84:6789/0 conn(0x7f426414be90 legacy=0x7f426414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: _finish_hunting 0 2024-03-25T18:15:00.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 monclient: found mon.noname-b 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640b1030 con 0x7f42640b4d60 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: _finish_auth 0 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: _check_auth_tickets 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4260001690 con 0x7f42640b4d60 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f426bc7b640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f426bc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f4268a15640 10 monclient: dump: 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:00.333 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.331+0000 7f426bc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:00.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42600017f0 con 0x7f42640b4d60 2024-03-25T18:15:00.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:00.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 1 -- 172.21.15.84:0/3682200961 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4260001c00 con 0x7f42640b4d60 2024-03-25T18:15:00.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f426bc7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f426bc7b640 4 set_mon_vals no callback set 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.347+0000 7f4268a15640 10 monclient: dump: 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:00.348 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 10 monclient: shutdown 2024-03-25T18:15:00.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:00.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 1 -- 172.21.15.84:0/3682200961 >> v1:172.21.15.84:6790/0 conn(0x7f42640b4d60 legacy=0x7f426413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 1 -- 172.21.15.84:0/3682200961 shutdown_connections 2024-03-25T18:15:00.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 1 -- 172.21.15.84:0/3682200961 >> 172.21.15.84:0/3682200961 conn(0x7f42640617a0 msgr2=0x7f4264061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 1 -- 172.21.15.84:0/3682200961 shutdown_connections 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.348+0000 7f426bc7b640 1 -- 172.21.15.84:0/3682200961 wait complete. 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.349+0000 7f426bc7b640 10 monclient: build_initial_monmap 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.349+0000 7f426bc7b640 10 monclient: monmap: 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.349+0000 7f426bc7b640 1 librados: starting msgr at 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.349+0000 7f426bc7b640 1 librados: starting objecter 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 Processor -- start 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 -- start start 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 librados: setting wanted keys 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 librados: calling monclient init 2024-03-25T18:15:00.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient: init 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-c con 0x7f42640b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-b con 0x7f42640b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient(hunting): picked mon.noname-a con 0x7f426414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4264141530 con 0x7f426414be90 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426413b7e0 con 0x7f42640b4d60 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4264140380 con 0x7f42640b1730 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:00.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.350+0000 7f426bc7b640 10 monclient(hunting): authenticate will time out at 7544.648882s 2024-03-25T18:15:00.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.356+0000 7f4269216640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f42640b4d60 0x7f42641de3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:43190/0 (socket says 172.21.15.84:43190) 2024-03-25T18:15:00.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.356+0000 7f4269216640 1 -- 172.21.15.84:0/4272706584 learned_addr learned my addr 172.21.15.84:0/4272706584 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:00.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.358+0000 7f425a7fc640 10 client.?.objecter ms_handle_connect 0x7f42640b4d60 2024-03-25T18:15:00.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.358+0000 7f425a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177889113 0 0) 0x7f426413b7e0 con 0x7f42640b4d60 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:15:00.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.365+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248004eb0 con 0x7f42640b4d60 2024-03-25T18:15:00.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.367+0000 7f425a7fc640 10 client.?.objecter ms_handle_connect 0x7f426414be90 2024-03-25T18:15:00.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.367+0000 7f425a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.369+0000 7f425a7fc640 10 client.?.objecter ms_handle_connect 0x7f42640b1730 2024-03-25T18:15:00.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.369+0000 7f425a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.379+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021898682 0 0) 0x7f4248004eb0 con 0x7f42640b4d60 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f426413b7e0 con 0x7f42640b4d60 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4254001a30 con 0x7f42640b4d60 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 10 monclient(hunting): dump: 2024-03-25T18:15:00.380 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.380+0000 7f425a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038170716 0 0) 0x7f4264141530 con 0x7f426414be90 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:15:00.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.382+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248004eb0 con 0x7f426414be90 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154112912 0 0) 0x7f4264140380 con 0x7f42640b1730 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:15:00.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.388+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4264141530 con 0x7f42640b1730 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3732258975 0 0) 0x7f426413b7e0 con 0x7f42640b4d60 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 >> v1:172.21.15.84:6790/0 conn(0x7f42640b1730 legacy=0x7f4264139100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 >> v1:172.21.15.84:6789/0 conn(0x7f426414be90 legacy=0x7f42641e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 1 monclient: found mon.b 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264140510 con 0x7f42640b4d60 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:00.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f425a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 10 monclient: _renew_subs 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4264139870 con 0x7f42640b4d60 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 10 monclient: _renew_subs 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f426413bee0 con 0x7f42640b4d60 2024-03-25T18:15:00.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.394+0000 7f426bc7b640 1 librados: init done 2024-03-25T18:15:00.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.395+0000 7f426bc7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:00.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.395+0000 7f426bc7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:00.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.395+0000 7f426bc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:00.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.395+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42640afbd0 con 0x7f42640b4d60 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4254001cd0 con 0x7f42640b4d60 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4254003a60 con 0x7f42640b4d60 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425a7fc640 10 monclient: dump: 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.671 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:00.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.670+0000 7f425bfff640 4 set_mon_vals no callback set 2024-03-25T18:15:00.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.671+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f4254003c40 con 0x7f42640b4d60 2024-03-25T18:15:00.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.672+0000 7f4269a17640 10 monclient: get_auth_request con 0x7f424806de80 auth_method 0 2024-03-25T18:15:00.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.672+0000 7f425a7fc640 10 client.4669.objecter ms_handle_connect 0x7f424806de80 2024-03-25T18:15:00.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f4254054930 con 0x7f42640b4d60 2024-03-25T18:15:00.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 10 client.4669.objecter ms_dispatch 0x7f4264000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:00.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:00.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:00.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:15:00.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4254050270 con 0x7f42640b4d60 2024-03-25T18:15:00.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:00.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.676+0000 7f425a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:00.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.920+0000 7f426bc7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:00.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.920+0000 7f426bc7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:00.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.920+0000 7f426bc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:00.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:00.920+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42640627b0 con 0x7f42640b4d60 2024-03-25T18:15:01.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.067+0000 7f425a7fc640 1 -- 172.21.15.84:0/4272706584 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f4254052370 con 0x7f42640b4d60 2024-03-25T18:15:01.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.067+0000 7f425a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.067+0000 7f425a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:01.068 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:01.069 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:01.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 10 librados: watch_flush enter 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 10 librados: watch_flush exit 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 >> v1:172.21.15.157:6800/1801259026 conn(0x7f424806de80 legacy=0x7f4248070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 10 monclient: shutdown 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.069+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 >> v1:172.21.15.157:6789/0 conn(0x7f42640b4d60 legacy=0x7f42641de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.070+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 shutdown_connections 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.070+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 >> 172.21.15.84:0/4272706584 conn(0x7f42640617a0 msgr2=0x7f42640b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.070+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 shutdown_connections 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.070+0000 7f426bc7b640 1 -- 172.21.15.84:0/4272706584 wait complete. 2024-03-25T18:15:01.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:01.070+0000 7f426bc7b640 1 librados: shutdown 2024-03-25T18:15:01.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:04.066 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:00.885 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.122 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:00.941 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.152 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:00.971 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.167 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:00.950 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.175 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:00.959 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.189 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:01.008 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.196 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:01.015 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.205 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:00.988 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:04.241 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:01.060 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:07.888 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.000+0000 7efc8ecd9640 10 monclient: get_monmap_and_config 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.000+0000 7efc8ecd9640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.000+0000 7efc8ecd9640 10 monclient: monmap: 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:08.002357+0000 2024-03-25T18:15:08.002 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:08.002357+0000 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:08.003 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.001+0000 7efc8ecd9640 1 Processor -- start 2024-03-25T18:15:08.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.001+0000 7efc8ecd9640 1 -- start start 2024-03-25T18:15:08.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.001+0000 7efc8ecd9640 10 monclient: init 2024-03-25T18:15:08.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-c con 0x7efc880b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-b con 0x7efc880b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-a con 0x7efc8814be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880af940 con 0x7efc8814be90 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880b0a70 con 0x7efc880b4d60 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880b0d50 con 0x7efc880b1730 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.002+0000 7efc8ecd9640 10 monclient(hunting): authenticate will time out at 7552.301626s 2024-03-25T18:15:08.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.011+0000 7efc87fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7efc880b1730 0x7efc880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56278/0 (socket says 172.21.15.84:56278) 2024-03-25T18:15:08.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.011+0000 7efc87fff640 1 -- 172.21.15.84:0/464080623 learned_addr learned my addr 172.21.15.84:0/464080623 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778067162 0 0) 0x7efc880b0d50 con 0x7efc880b1730 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:15:08.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.029+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc78003520 con 0x7efc880b1730 2024-03-25T18:15:08.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511839416 0 0) 0x7efc880af940 con 0x7efc8814be90 2024-03-25T18:15:08.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:15:08.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.041+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc880b0d50 con 0x7efc8814be90 2024-03-25T18:15:08.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1668829033 0 0) 0x7efc78003520 con 0x7efc880b1730 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 >> v1:172.21.15.157:6789/0 conn(0x7efc880b4d60 legacy=0x7efc8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 >> v1:172.21.15.84:6789/0 conn(0x7efc8814be90 legacy=0x7efc8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 monclient: found mon.noname-b 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880b1030 con 0x7efc880b1730 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efc74001690 con 0x7efc880b1730 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc8ecd9640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc8ecd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc877fe640 10 monclient: dump: 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.049+0000 7efc8ecd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc740017f0 con 0x7efc880b1730 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 1 -- 172.21.15.84:0/464080623 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efc74001c00 con 0x7efc880b1730 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc877fe640 10 monclient: dump: 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:08.066 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:08.067 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc8ecd9640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:08.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.064+0000 7efc8ecd9640 4 set_mon_vals no callback set 2024-03-25T18:15:08.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 10 monclient: shutdown 2024-03-25T18:15:08.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:08.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 1 -- 172.21.15.84:0/464080623 >> v1:172.21.15.84:6790/0 conn(0x7efc880b1730 legacy=0x7efc880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 1 -- 172.21.15.84:0/464080623 shutdown_connections 2024-03-25T18:15:08.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 1 -- 172.21.15.84:0/464080623 >> 172.21.15.84:0/464080623 conn(0x7efc880617a0 msgr2=0x7efc88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 1 -- 172.21.15.84:0/464080623 shutdown_connections 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.065+0000 7efc8ecd9640 1 -- 172.21.15.84:0/464080623 wait complete. 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 10 monclient: monmap: 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:08.069 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 librados: starting msgr at 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 librados: starting objecter 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 Processor -- start 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 -- start start 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 librados: setting wanted keys 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 1 librados: calling monclient init 2024-03-25T18:15:08.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.066+0000 7efc8ecd9640 10 monclient: init 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-b con 0x7efc880b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-a con 0x7efc880b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient(hunting): picked mon.noname-c con 0x7efc8814be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc88140740 con 0x7efc880b4d60 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc88140380 con 0x7efc880b1730 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc88140510 con 0x7efc8814be90 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.067+0000 7efc8ecd9640 10 monclient(hunting): authenticate will time out at 7552.366771s 2024-03-25T18:15:08.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.080+0000 7efc87fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7efc880b4d60 0x7efc881daca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49666/0 (socket says 172.21.15.84:49666) 2024-03-25T18:15:08.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.080+0000 7efc87fff640 1 -- 172.21.15.84:0/4055222525 learned_addr learned my addr 172.21.15.84:0/4055222525 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:08.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.083+0000 7efc857fa640 10 client.?.objecter ms_handle_connect 0x7efc8814be90 2024-03-25T18:15:08.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.083+0000 7efc857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.084+0000 7efc857fa640 10 client.?.objecter ms_handle_connect 0x7efc880b1730 2024-03-25T18:15:08.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.084+0000 7efc857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.087+0000 7efc857fa640 10 client.?.objecter ms_handle_connect 0x7efc880b4d60 2024-03-25T18:15:08.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.087+0000 7efc857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476678491 0 0) 0x7efc88140740 con 0x7efc880b4d60 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:15:08.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.100+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6c004eb0 con 0x7efc880b4d60 2024-03-25T18:15:08.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878339172 0 0) 0x7efc88140510 con 0x7efc8814be90 2024-03-25T18:15:08.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:15:08.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.101+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88140740 con 0x7efc8814be90 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138293330 0 0) 0x7efc88140380 con 0x7efc880b1730 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:15:08.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.112+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88140510 con 0x7efc880b1730 2024-03-25T18:15:08.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199060865 0 0) 0x7efc6c004eb0 con 0x7efc880b4d60 2024-03-25T18:15:08.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc88140380 con 0x7efc880b4d60 2024-03-25T18:15:08.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efc74001bc0 con 0x7efc880b4d60 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 10 monclient(hunting): dump: 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.114+0000 7efc857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1749619351 0 0) 0x7efc88140740 con 0x7efc8814be90 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc6c004eb0 con 0x7efc8814be90 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efc80001b90 con 0x7efc8814be90 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.117+0000 7efc857fa640 10 monclient(hunting): dump: 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.119 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:08.120 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3297269829 0 0) 0x7efc88140380 con 0x7efc880b4d60 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 >> v1:172.21.15.84:6790/0 conn(0x7efc8814be90 legacy=0x7efc881db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 >> v1:172.21.15.157:6789/0 conn(0x7efc880b1730 legacy=0x7efc88134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 1 monclient: found mon.a 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc88137400 con 0x7efc880b4d60 2024-03-25T18:15:08.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 10 monclient: _renew_subs 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc88137610 con 0x7efc880b4d60 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 10 monclient: _renew_subs 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc88137f80 con 0x7efc880b4d60 2024-03-25T18:15:08.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.129+0000 7efc8ecd9640 1 librados: init done 2024-03-25T18:15:08.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.130+0000 7efc8ecd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.130+0000 7efc8ecd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.130+0000 7efc8ecd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:08.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.130+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc881401f0 con 0x7efc880b4d60 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc74004a90 con 0x7efc880b4d60 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7efc74004da0 con 0x7efc880b4d60 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc857fa640 10 monclient: dump: 2024-03-25T18:15:08.457 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:08.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.455+0000 7efc86ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:08.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.456+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7efc7401ad70 con 0x7efc880b4d60 2024-03-25T18:15:08.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.457+0000 7efc8ca75640 10 monclient: get_auth_request con 0x7efc6c06df70 auth_method 0 2024-03-25T18:15:08.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.457+0000 7efc857fa640 10 client.4677.objecter ms_handle_connect 0x7efc6c06df70 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.461+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7efc740548d0 con 0x7efc880b4d60 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.461+0000 7efc857fa640 10 client.4677.objecter ms_dispatch 0x7efc88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.461+0000 7efc857fa640 3 client.4677.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.461+0000 7efc857fa640 3 client.4677.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.462+0000 7efc857fa640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:15:08.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.462+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc740503e0 con 0x7efc880b4d60 2024-03-25T18:15:08.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.462+0000 7efc857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.462+0000 7efc857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.705+0000 7efc8ecd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.705+0000 7efc8ecd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.705+0000 7efc8ecd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.705+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc880627b0 con 0x7efc880b4d60 2024-03-25T18:15:08.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.831+0000 7efc857fa640 1 -- 172.21.15.84:0/4055222525 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7efc74052310 con 0x7efc880b4d60 2024-03-25T18:15:08.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.831+0000 7efc857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:08.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.831+0000 7efc857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:08.833 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:08.833 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 10 librados: watch_flush enter 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 10 librados: watch_flush exit 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 >> v1:172.21.15.157:6800/1801259026 conn(0x7efc6c06df70 legacy=0x7efc6c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 10 monclient: shutdown 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 >> v1:172.21.15.84:6789/0 conn(0x7efc880b4d60 legacy=0x7efc881daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 shutdown_connections 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 >> 172.21.15.84:0/4055222525 conn(0x7efc880617a0 msgr2=0x7efc880b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 shutdown_connections 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.833+0000 7efc8ecd9640 1 -- 172.21.15.84:0/4055222525 wait complete. 2024-03-25T18:15:08.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:08.834+0000 7efc8ecd9640 1 librados: shutdown 2024-03-25T18:15:08.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:15.654 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.772+0000 7f235283e640 10 monclient: get_monmap_and_config 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.772+0000 7f235283e640 10 monclient: build_initial_monmap 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.772+0000 7f235283e640 10 monclient: monmap: 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:15.774030+0000 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:15.774030+0000 2024-03-25T18:15:15.774 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:15.775 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:15.775 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:15.775 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:15.775 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:15.775 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.773+0000 7f235283e640 1 Processor -- start 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.773+0000 7f235283e640 1 -- start start 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.773+0000 7f235283e640 10 monclient: init 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient(hunting): picked mon.noname-c con 0x7f234c0a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient(hunting): picked mon.noname-b con 0x7f234c13f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient(hunting): picked mon.noname-a con 0x7f234c1430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:15.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c142520 con 0x7f234c1430c0 2024-03-25T18:15:15.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0b28a0 con 0x7f234c0a98f0 2024-03-25T18:15:15.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0b2a30 con 0x7f234c13f710 2024-03-25T18:15:15.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:15.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.774+0000 7f235283e640 10 monclient(hunting): authenticate will time out at 7560.073068s 2024-03-25T18:15:15.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.783+0000 7f234bfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f234c0a98f0 0x7f234c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:46028/0 (socket says 172.21.15.84:46028) 2024-03-25T18:15:15.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.783+0000 7f234bfff640 1 -- 172.21.15.84:0/4162495844 learned_addr learned my addr 172.21.15.84:0/4162495844 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:15.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666533341 0 0) 0x7f234c142520 con 0x7f234c1430c0 2024-03-25T18:15:15.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:15:15.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.795+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2334003520 con 0x7f234c1430c0 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048519232 0 0) 0x7f234c0b2a30 con 0x7f234c13f710 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:15:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.798+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c142520 con 0x7f234c13f710 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239061576 0 0) 0x7f234c0b28a0 con 0x7f234c0a98f0 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:15:15.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.803+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c0b2a30 con 0x7f234c0a98f0 2024-03-25T18:15:15.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4273769822 0 0) 0x7f2334003520 con 0x7f234c1430c0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 >> v1:172.21.15.84:6790/0 conn(0x7f234c13f710 legacy=0x7f234c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 >> v1:172.21.15.157:6789/0 conn(0x7f234c0a98f0 legacy=0x7f234c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 monclient: found mon.noname-a 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0b44c0 con 0x7f234c1430c0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2344001a10 con 0x7f234c1430c0 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f235283e640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f234affd640 10 monclient: dump: 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.807+0000 7f235283e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:15.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2344001b70 con 0x7f234c1430c0 2024-03-25T18:15:15.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:15.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 1 -- 172.21.15.84:0/4162495844 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2344001f80 con 0x7f234c1430c0 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f234affd640 10 monclient: dump: 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f235283e640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:15.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.829+0000 7f235283e640 4 set_mon_vals no callback set 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 10 monclient: shutdown 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 1 -- 172.21.15.84:0/4162495844 >> v1:172.21.15.84:6789/0 conn(0x7f234c1430c0 legacy=0x7f234c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 1 -- 172.21.15.84:0/4162495844 shutdown_connections 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 1 -- 172.21.15.84:0/4162495844 >> 172.21.15.84:0/4162495844 conn(0x7f234c0b7ea0 msgr2=0x7f234c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 1 -- 172.21.15.84:0/4162495844 shutdown_connections 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.830+0000 7f235283e640 1 -- 172.21.15.84:0/4162495844 wait complete. 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.831+0000 7f235283e640 10 monclient: build_initial_monmap 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.831+0000 7f235283e640 10 monclient: monmap: 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:15.833 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.831+0000 7f235283e640 1 librados: starting msgr at 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.831+0000 7f235283e640 1 librados: starting objecter 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.831+0000 7f235283e640 1 Processor -- start 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 -- start start 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 librados: setting wanted keys 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 librados: calling monclient init 2024-03-25T18:15:15.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient: init 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient(hunting): picked mon.noname-c con 0x7f234c0a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient(hunting): picked mon.noname-a con 0x7f234c13f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient(hunting): picked mon.noname-b con 0x7f234c1430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:15.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c136c50 con 0x7f234c13f710 2024-03-25T18:15:15.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c136ec0 con 0x7f234c1430c0 2024-03-25T18:15:15.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c1317f0 con 0x7f234c0a98f0 2024-03-25T18:15:15.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:15.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.832+0000 7f235283e640 10 monclient(hunting): authenticate will time out at 7560.131657s 2024-03-25T18:15:15.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.843+0000 7f2350ddb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f234c1430c0 0x7f234c1db380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:46086/0 (socket says 172.21.15.84:46086) 2024-03-25T18:15:15.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.843+0000 7f2350ddb640 1 -- 172.21.15.84:0/2197390960 learned_addr learned my addr 172.21.15.84:0/2197390960 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:15.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.851+0000 7f2348ff9640 10 client.?.objecter ms_handle_connect 0x7f234c1430c0 2024-03-25T18:15:15.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.851+0000 7f2348ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:15.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.860+0000 7f2348ff9640 10 client.?.objecter ms_handle_connect 0x7f234c0a98f0 2024-03-25T18:15:15.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.860+0000 7f2348ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:15.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.866+0000 7f2348ff9640 10 client.?.objecter ms_handle_connect 0x7f234c13f710 2024-03-25T18:15:15.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.866+0000 7f2348ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134024709 0 0) 0x7f234c1317f0 con 0x7f234c0a98f0 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:15:15.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.874+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2338004eb0 con 0x7f234c0a98f0 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392959943 0 0) 0x7f234c136ec0 con 0x7f234c1430c0 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:15:15.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.877+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c1317f0 con 0x7f234c1430c0 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319383482 0 0) 0x7f234c136c50 con 0x7f234c13f710 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:15:15.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.883+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c136ec0 con 0x7f234c13f710 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535704099 0 0) 0x7f234c1317f0 con 0x7f234c1430c0 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f234c136c50 con 0x7f234c1430c0 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2344001e80 con 0x7f234c1430c0 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:15.891 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662762097 0 0) 0x7f2338004eb0 con 0x7f234c0a98f0 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f234c1317f0 con 0x7f234c0a98f0 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f233c0018f0 con 0x7f234c0a98f0 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.889+0000 7f2348ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:15.892 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:15.893 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:15.893 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:15.893 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:15.893 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:15.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.897+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254009063 0 0) 0x7f234c136c50 con 0x7f234c1430c0 2024-03-25T18:15:15.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 >> v1:172.21.15.84:6790/0 conn(0x7f234c0a98f0 legacy=0x7f234c1310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 >> v1:172.21.15.84:6789/0 conn(0x7f234c13f710 legacy=0x7f234c1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 1 monclient: found mon.b 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c131980 con 0x7f234c1430c0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f2348ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 10 monclient: _renew_subs 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c135070 con 0x7f234c1430c0 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 10 monclient: _renew_subs 2024-03-25T18:15:15.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c0b7870 con 0x7f234c1430c0 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.898+0000 7f235283e640 1 librados: init done 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.899+0000 7f235283e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.899+0000 7f235283e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.899+0000 7f235283e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:15.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:15.899+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c135c70 con 0x7f234c1430c0 2024-03-25T18:15:16.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2344001590 con 0x7f234c1430c0 2024-03-25T18:15:16.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2344003c20 con 0x7f234c1430c0 2024-03-25T18:15:16.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f2348ff9640 10 monclient: dump: 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.179+0000 7f234a7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.180+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2344003e00 con 0x7f234c1430c0 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.181+0000 7f234bfff640 10 monclient: get_auth_request con 0x7f233806df70 auth_method 0 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.181+0000 7f2348ff9640 10 client.4681.objecter ms_handle_connect 0x7f233806df70 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.185+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f23440047c0 con 0x7f234c1430c0 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.185+0000 7f2348ff9640 10 client.4681.objecter ms_dispatch 0x7f234c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.185+0000 7f2348ff9640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.185+0000 7f2348ff9640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.185+0000 7f2348ff9640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.186+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2344050430 con 0x7f234c1430c0 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.186+0000 7f2348ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.186+0000 7f2348ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:16.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.411+0000 7f235283e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.411+0000 7f235283e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.411+0000 7f235283e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:16.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.411+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f234c0627b0 con 0x7f234c1430c0 2024-03-25T18:15:16.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.435+0000 7f2348ff9640 1 -- 172.21.15.84:0/2197390960 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f2344052530 con 0x7f234c1430c0 2024-03-25T18:15:16.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.435+0000 7f2348ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.435+0000 7f2348ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:16.437 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:16.438 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.437+0000 7f235283e640 10 librados: watch_flush enter 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 10 librados: watch_flush exit 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 >> v1:172.21.15.157:6800/1801259026 conn(0x7f233806df70 legacy=0x7f2338070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 10 monclient: shutdown 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 >> v1:172.21.15.157:6789/0 conn(0x7f234c1430c0 legacy=0x7f234c1db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 shutdown_connections 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 >> 172.21.15.84:0/2197390960 conn(0x7f234c0b7ea0 msgr2=0x7f234c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 shutdown_connections 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 -- 172.21.15.84:0/2197390960 wait complete. 2024-03-25T18:15:16.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:16.438+0000 7f235283e640 1 librados: shutdown 2024-03-25T18:15:16.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:23.259 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:23.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.374+0000 7f64b5b65640 10 monclient: get_monmap_and_config 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.374+0000 7f64b5b65640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.374+0000 7f64b5b65640 10 monclient: monmap: 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:23.375935+0000 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:23.375935+0000 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:23.376 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.375+0000 7f64b5b65640 1 Processor -- start 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.375+0000 7f64b5b65640 1 -- start start 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.375+0000 7f64b5b65640 10 monclient: init 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-b con 0x7f64b006ab80 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-c con 0x7f64b006af60 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-a con 0x7f64b00b00b0 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00b0500 con 0x7f64b00b00b0 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b013ffa0 con 0x7f64b006af60 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00af520 con 0x7f64b006ab80 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.376+0000 7f64b5b65640 10 monclient(hunting): authenticate will time out at 7567.675263s 2024-03-25T18:15:23.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.384+0000 7f64af7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f64b006ab80 0x7f64b00ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38052/0 (socket says 172.21.15.84:38052) 2024-03-25T18:15:23.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.384+0000 7f64af7fe640 1 -- 172.21.15.84:0/685191590 learned_addr learned my addr 172.21.15.84:0/685191590 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:23.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298027446 0 0) 0x7f64b00b0500 con 0x7f64b00b00b0 2024-03-25T18:15:23.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:15:23.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.404+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498003520 con 0x7f64b00b00b0 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986973605 0 0) 0x7f64b013ffa0 con 0x7f64b006af60 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:15:23.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.408+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b00b0500 con 0x7f64b006af60 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459961508 0 0) 0x7f64b00af520 con 0x7f64b006ab80 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:15:23.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.409+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b013ffa0 con 0x7f64b006ab80 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460557533 0 0) 0x7f6498003520 con 0x7f64b00b00b0 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 >> v1:172.21.15.84:6790/0 conn(0x7f64b006ab80 legacy=0x7f64b00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 >> v1:172.21.15.157:6789/0 conn(0x7f64b006af60 legacy=0x7f64b013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0072890 con 0x7f64b00b00b0 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:23.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f64a8001a10 con 0x7f64b00b00b0 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64b5b65640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64b5b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64ae7fc640 10 monclient: dump: 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:23.415 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:23.416 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.412+0000 7f64b5b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a8001b70 con 0x7f64b00b00b0 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 1 -- 172.21.15.84:0/685191590 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f64a8001f80 con 0x7f64b00b00b0 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64b5b65640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64b5b65640 4 set_mon_vals no callback set 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.433+0000 7f64ae7fc640 10 monclient: dump: 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.435 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.436 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.436 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:23.436 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:23.436 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:23.436 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 10 monclient: shutdown 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 1 -- 172.21.15.84:0/685191590 >> v1:172.21.15.84:6789/0 conn(0x7f64b00b00b0 legacy=0x7f64b014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 1 -- 172.21.15.84:0/685191590 shutdown_connections 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 1 -- 172.21.15.84:0/685191590 >> 172.21.15.84:0/685191590 conn(0x7f64b00b4a80 msgr2=0x7f64b00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:23.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 1 -- 172.21.15.84:0/685191590 shutdown_connections 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.434+0000 7f64b5b65640 1 -- 172.21.15.84:0/685191590 wait complete. 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.435+0000 7f64b5b65640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.435+0000 7f64b5b65640 10 monclient: monmap: 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.435+0000 7f64b5b65640 1 librados: starting msgr at 2024-03-25T18:15:23.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.435+0000 7f64b5b65640 1 librados: starting objecter 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.435+0000 7f64b5b65640 1 Processor -- start 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 -- start start 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 librados: setting wanted keys 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 librados: calling monclient init 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient: init 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-c con 0x7f64b006ab80 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-b con 0x7f64b00b00b0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient(hunting): picked mon.noname-a con 0x7f64b0143460 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b011d9e0 con 0x7f64b0143460 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b011dc50 con 0x7f64b00b00b0 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b011df30 con 0x7f64b006ab80 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.436+0000 7f64b5b65640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.437+0000 7f64b5b65640 10 monclient(hunting): authenticate will time out at 7567.735895s 2024-03-25T18:15:23.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.444+0000 7f64aeffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f64b00b00b0 0x7f64b01db030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:41744/0 (socket says 172.21.15.84:41744) 2024-03-25T18:15:23.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.444+0000 7f64aeffd640 1 -- 172.21.15.84:0/1963868156 learned_addr learned my addr 172.21.15.84:0/1963868156 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:23.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.445+0000 7f6497fff640 10 client.?.objecter ms_handle_connect 0x7f64b00b00b0 2024-03-25T18:15:23.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.445+0000 7f6497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.450+0000 7f6497fff640 10 client.?.objecter ms_handle_connect 0x7f64b006ab80 2024-03-25T18:15:23.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.450+0000 7f6497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.451+0000 7f6497fff640 10 client.?.objecter ms_handle_connect 0x7f64b0143460 2024-03-25T18:15:23.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.451+0000 7f6497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326407213 0 0) 0x7f64b011dc50 con 0x7f64b00b00b0 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:15:23.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.458+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f649c004eb0 con 0x7f64b00b00b0 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353289686 0 0) 0x7f64b011d9e0 con 0x7f64b0143460 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:15:23.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.460+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b011dc50 con 0x7f64b0143460 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488348702 0 0) 0x7f64b011df30 con 0x7f64b006ab80 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:15:23.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.467+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b011d9e0 con 0x7f64b006ab80 2024-03-25T18:15:23.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170796501 0 0) 0x7f649c004eb0 con 0x7f64b00b00b0 2024-03-25T18:15:23.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b011df30 con 0x7f64b00b00b0 2024-03-25T18:15:23.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f64a4001a30 con 0x7f64b00b00b0 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 10 monclient(hunting): dump: 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.471+0000 7f6497fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:23.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109393481 0 0) 0x7f64b011dc50 con 0x7f64b0143460 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f649c004eb0 con 0x7f64b0143460 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f64a8001e80 con 0x7f64b0143460 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.472+0000 7f6497fff640 10 monclient(hunting): dump: 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:23.475 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2434629390 0 0) 0x7f64b011df30 con 0x7f64b00b00b0 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 >> v1:172.21.15.84:6790/0 conn(0x7f64b006ab80 legacy=0x7f64b0145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 >> v1:172.21.15.84:6789/0 conn(0x7f64b0143460 legacy=0x7f64b013ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 1 monclient: found mon.b 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:23.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b01dab10 con 0x7f64b00b00b0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f6497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 10 monclient: _renew_subs 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0146510 con 0x7f64b00b00b0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 10 monclient: _renew_subs 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b013b270 con 0x7f64b00b00b0 2024-03-25T18:15:23.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.487+0000 7f64b5b65640 1 librados: init done 2024-03-25T18:15:23.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.488+0000 7f64b5b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:23.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.488+0000 7f64b5b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:23.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.488+0000 7f64b5b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:23.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.488+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b01da970 con 0x7f64b00b00b0 2024-03-25T18:15:23.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.494+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a4001c70 con 0x7f64b00b00b0 2024-03-25T18:15:23.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.494+0000 7f6497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:23.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.494+0000 7f64adffb640 4 set_mon_vals no callback set 2024-03-25T18:15:23.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.766+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f64a40038c0 con 0x7f64b00b00b0 2024-03-25T18:15:23.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.766+0000 7f6497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.766+0000 7f6497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:23.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.766+0000 7f6497fff640 10 monclient: dump: 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:23.768 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:23.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.767+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f64a4003aa0 con 0x7f64b00b00b0 2024-03-25T18:15:23.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.768+0000 7f64af7fe640 10 monclient: get_auth_request con 0x7f649c06df70 auth_method 0 2024-03-25T18:15:23.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.768+0000 7f6497fff640 10 client.4687.objecter ms_handle_connect 0x7f649c06df70 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.771+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4206527539 0 0) 0x7f64a40546b0 con 0x7f64b00b00b0 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.771+0000 7f6497fff640 10 client.4687.objecter ms_dispatch 0x7f64b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.771+0000 7f6497fff640 3 client.4687.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.771+0000 7f6497fff640 3 client.4687.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.771+0000 7f6497fff640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:15:23.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.774+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64a40501c0 con 0x7f64b00b00b0 2024-03-25T18:15:23.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.774+0000 7f6497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:23.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:23.774+0000 7f6497fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:24.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.028+0000 7f64b5b65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.028+0000 7f64b5b65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.028+0000 7f64b5b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:24.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.028+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b01daca0 con 0x7f64b00b00b0 2024-03-25T18:15:24.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.164+0000 7f6497fff640 1 -- 172.21.15.84:0/1963868156 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1433441218) 0x7f64a40520f0 con 0x7f64b00b00b0 2024-03-25T18:15:24.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.164+0000 7f6497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.164+0000 7f6497fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:24.166 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:24.166 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:05:25.146540+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:05:25.146540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:24.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 10 librados: watch_flush enter 2024-03-25T18:15:24.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 10 librados: watch_flush exit 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 >> v1:172.21.15.157:6800/1801259026 conn(0x7f649c06df70 legacy=0x7f649c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 10 monclient: shutdown 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 >> v1:172.21.15.157:6789/0 conn(0x7f64b00b00b0 legacy=0x7f64b01db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 shutdown_connections 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 >> 172.21.15.84:0/1963868156 conn(0x7f64b00b4a80 msgr2=0x7f64b00b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 shutdown_connections 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.166+0000 7f64b5b65640 1 -- 172.21.15.84:0/1963868156 wait complete. 2024-03-25T18:15:24.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:24.167+0000 7f64b5b65640 1 librados: shutdown 2024-03-25T18:15:24.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:24.397 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:21.215 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.411 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:21.230 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.448 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:21.267 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.475 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:21.294 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.480 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:21.299 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.497 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:21.281 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.505 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:21.288 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.514 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:21.333 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.531 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:21.314 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.986 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:31.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.095+0000 7f1dc110e640 10 monclient: get_monmap_and_config 2024-03-25T18:15:31.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.095+0000 7f1dc110e640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.095+0000 7f1dc110e640 10 monclient: monmap: 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:31.096837+0000 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:31.096837+0000 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:31.097 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.096+0000 7f1dc110e640 1 Processor -- start 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.096+0000 7f1dc110e640 1 -- start start 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.096+0000 7f1dc110e640 10 monclient: init 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-b con 0x7f1dbc0b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:31.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-a con 0x7f1dbc0b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-c con 0x7f1dbc14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc0af940 con 0x7f1dbc0b1730 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc0b0a70 con 0x7f1dbc14be90 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc0b0d50 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.097+0000 7f1dc110e640 10 monclient(hunting): authenticate will time out at 7575.396024s 2024-03-25T18:15:31.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.111+0000 7f1dbad9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1dbc0b4d60 0x7f1dbc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59780/0 (socket says 172.21.15.84:59780) 2024-03-25T18:15:31.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.111+0000 7f1dbad9d640 1 -- 172.21.15.84:0/4192828831 learned_addr learned my addr 172.21.15.84:0/4192828831 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:31.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817886402 0 0) 0x7f1dbc0b0a70 con 0x7f1dbc14be90 2024-03-25T18:15:31.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4003520 con 0x7f1dbc14be90 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040695655 0 0) 0x7f1dbc0b0d50 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:15:31.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.129+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dbc0b0a70 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.132+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270621833 0 0) 0x7f1dbc0af940 con 0x7f1dbc0b1730 2024-03-25T18:15:31.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.132+0000 7f1db9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.132+0000 7f1db9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.132+0000 7f1db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.132+0000 7f1db9d9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:15:31.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.133+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dbc0b0d50 con 0x7f1dbc0b1730 2024-03-25T18:15:31.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.143+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703549953 0 0) 0x7f1dbc0b0a70 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 >> v1:172.21.15.157:6789/0 conn(0x7f1dbc14be90 legacy=0x7f1dbc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 >> v1:172.21.15.84:6789/0 conn(0x7f1dbc0b1730 legacy=0x7f1dbc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 1 monclient: found mon.noname-b 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc0b1030 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1dac001670 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1db9d9b640 10 monclient: dump: 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.146 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1dc110e640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:15:31.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.144+0000 7f1dc110e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dac0017d0 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 1 -- 172.21.15.84:0/4192828831 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1dac001be0 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1dc110e640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1dc110e640 4 set_mon_vals no callback set 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.161+0000 7f1db9d9b640 10 monclient: dump: 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.163 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:31.164 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 10 monclient: shutdown 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 1 -- 172.21.15.84:0/4192828831 >> v1:172.21.15.84:6790/0 conn(0x7f1dbc0b4d60 legacy=0x7f1dbc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 1 -- 172.21.15.84:0/4192828831 shutdown_connections 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 1 -- 172.21.15.84:0/4192828831 >> 172.21.15.84:0/4192828831 conn(0x7f1dbc0617a0 msgr2=0x7f1dbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:31.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 1 -- 172.21.15.84:0/4192828831 shutdown_connections 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.162+0000 7f1dc110e640 1 -- 172.21.15.84:0/4192828831 wait complete. 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.163+0000 7f1dc110e640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.163+0000 7f1dc110e640 10 monclient: monmap: 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 librados: starting msgr at 2024-03-25T18:15:31.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 librados: starting objecter 2024-03-25T18:15:31.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 Processor -- start 2024-03-25T18:15:31.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 -- start start 2024-03-25T18:15:31.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 librados: setting wanted keys 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 1 librados: calling monclient init 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.164+0000 7f1dc110e640 10 monclient: init 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-c con 0x7f1dbc0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-a con 0x7f1dbc0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient(hunting): picked mon.noname-b con 0x7f1dbc14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc13e200 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc12f5f0 con 0x7f1dbc14be90 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc12f780 con 0x7f1dbc0b1730 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.165+0000 7f1dc110e640 10 monclient(hunting): authenticate will time out at 7575.464480s 2024-03-25T18:15:31.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.171+0000 7f1dbb59e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1dbc14be90 0x7f1dbc1379b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:36378/0 (socket says 172.21.15.84:36378) 2024-03-25T18:15:31.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.171+0000 7f1dbb59e640 1 -- 172.21.15.84:0/3150664027 learned_addr learned my addr 172.21.15.84:0/3150664027 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:31.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.172+0000 7f1da37fe640 10 client.?.objecter ms_handle_connect 0x7f1dbc14be90 2024-03-25T18:15:31.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.172+0000 7f1da37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975498487 0 0) 0x7f1dbc12f5f0 con 0x7f1dbc14be90 2024-03-25T18:15:31.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:15:31.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.178+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da8004eb0 con 0x7f1dbc14be90 2024-03-25T18:15:31.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.183+0000 7f1da37fe640 10 client.?.objecter ms_handle_connect 0x7f1dbc0b1730 2024-03-25T18:15:31.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.183+0000 7f1da37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518213512 0 0) 0x7f1da8004eb0 con 0x7f1dbc14be90 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dbc12f5f0 con 0x7f1dbc14be90 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1db4001c30 con 0x7f1dbc14be90 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 monclient(hunting): dump: 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.190 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 client.?.objecter ms_handle_connect 0x7f1dbc0b4d60 2024-03-25T18:15:31.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.188+0000 7f1da37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608572639 0 0) 0x7f1dbc12f780 con 0x7f1dbc0b1730 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:15:31.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.190+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da8004eb0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820025333 0 0) 0x7f1dbc13e200 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:15:31.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.194+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dbc12f780 con 0x7f1dbc0b4d60 2024-03-25T18:15:31.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128808914 0 0) 0x7f1da8004eb0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dbc13e200 con 0x7f1dbc0b1730 2024-03-25T18:15:31.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1dac001ae0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.202+0000 7f1da37fe640 10 monclient(hunting): dump: 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:31.204 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 424395789 0 0) 0x7f1dbc13e200 con 0x7f1dbc0b1730 2024-03-25T18:15:31.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 >> v1:172.21.15.157:6789/0 conn(0x7f1dbc14be90 legacy=0x7f1dbc1379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 >> v1:172.21.15.84:6789/0 conn(0x7f1dbc0b4d60 legacy=0x7f1dbc12eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 1 monclient: found mon.c 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc13dd20 con 0x7f1dbc0b1730 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1da37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 10 monclient: _renew_subs 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc13df70 con 0x7f1dbc0b1730 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 10 monclient: _renew_subs 2024-03-25T18:15:31.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc135f50 con 0x7f1dbc0b1730 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.214+0000 7f1dc110e640 1 librados: init done 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.215+0000 7f1dc110e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.215+0000 7f1dc110e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.215+0000 7f1dc110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.215+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc061ca0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dac001210 con 0x7f1dbc0b1730 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1dac004c30 con 0x7f1dbc0b1730 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 10 monclient: dump: 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:31.553 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1db959a640 4 set_mon_vals no callback set 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.551+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1dac01ac00 con 0x7f1dbc0b1730 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.552+0000 7f1dba59c640 10 monclient: get_auth_request con 0x7f1da806df80 auth_method 0 2024-03-25T18:15:31.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.552+0000 7f1da37fe640 10 client.4685.objecter ms_handle_connect 0x7f1da806df80 2024-03-25T18:15:31.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f1dac054870 con 0x7f1dbc0b1730 2024-03-25T18:15:31.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 10 client.4685.objecter ms_dispatch 0x7f1dbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:31.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:31.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:31.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-25T18:15:31.560 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dac0501b0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.560 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.560 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.558+0000 7f1da37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:31.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.771+0000 7f1dc110e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:31.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.771+0000 7f1dc110e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:31.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.771+0000 7f1dc110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:31.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.771+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dbc13e390 con 0x7f1dbc0b1730 2024-03-25T18:15:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.793+0000 7f1da37fe640 1 -- 172.21.15.84:0/3150664027 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f1dac0522b0 con 0x7f1dbc0b1730 2024-03-25T18:15:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.793+0000 7f1da37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.793+0000 7f1da37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:31.795 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:31.795 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:31.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 10 librados: watch_flush enter 2024-03-25T18:15:31.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 10 librados: watch_flush exit 2024-03-25T18:15:31.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 10 client.4685.objecter successfully canceled tick 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1da806df80 legacy=0x7f1da8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 10 monclient: shutdown 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 >> v1:172.21.15.84:6790/0 conn(0x7f1dbc0b1730 legacy=0x7f1dbc145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 shutdown_connections 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.795+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 >> 172.21.15.84:0/3150664027 conn(0x7f1dbc0617a0 msgr2=0x7f1dbc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.796+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 shutdown_connections 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.796+0000 7f1dc110e640 1 -- 172.21.15.84:0/3150664027 wait complete. 2024-03-25T18:15:31.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:31.796+0000 7f1dc110e640 1 librados: shutdown 2024-03-25T18:15:31.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:38.616 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:38.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.728+0000 7f2411b47640 10 monclient: get_monmap_and_config 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.728+0000 7f2411b47640 10 monclient: build_initial_monmap 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.728+0000 7f2411b47640 10 monclient: monmap: 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:38.729706+0000 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:38.729706+0000 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:38.730 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 1 Processor -- start 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 1 -- start start 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient: init 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-c con 0x7f240c0b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-a con 0x7f240c0b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-b con 0x7f240c14be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0af940 con 0x7f240c0b1730 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0b0a70 con 0x7f240c0b4d60 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0b0d50 con 0x7f240c14be90 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.729+0000 7f2411b47640 10 monclient(hunting): authenticate will time out at 7583.028883s 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.730+0000 7f240b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f240c0b4d60 0x7f240c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:46612/0 (socket says 172.21.15.84:46612) 2024-03-25T18:15:38.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.730+0000 7f240b7fe640 1 -- 172.21.15.84:0/4001956005 learned_addr learned my addr 172.21.15.84:0/4001956005 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767246063 0 0) 0x7f240c0b0a70 con 0x7f240c0b4d60 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:15:38.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.740+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f4003520 con 0x7f240c0b4d60 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070992587 0 0) 0x7f23f4003520 con 0x7f240c0b4d60 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 >> v1:172.21.15.84:6790/0 conn(0x7f240c14be90 legacy=0x7f240c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 >> v1:172.21.15.84:6789/0 conn(0x7f240c0b1730 legacy=0x7f240c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:15:38.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0b1030 con 0x7f240c0b4d60 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f23fc001690 con 0x7f240c0b4d60 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f240a7fc640 10 monclient: dump: 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:38.749 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:38.750 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:38.750 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f2411b47640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:15:38.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.746+0000 7f2411b47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23fc0017f0 con 0x7f240c0b4d60 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 1 -- 172.21.15.84:0/4001956005 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f23fc001c40 con 0x7f240c0b4d60 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f2411b47640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f2411b47640 4 set_mon_vals no callback set 2024-03-25T18:15:38.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.760+0000 7f240a7fc640 10 monclient: dump: 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:38.762 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 10 monclient: shutdown 2024-03-25T18:15:38.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:38.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 1 -- 172.21.15.84:0/4001956005 >> v1:172.21.15.157:6789/0 conn(0x7f240c0b4d60 legacy=0x7f240c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:38.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 1 -- 172.21.15.84:0/4001956005 shutdown_connections 2024-03-25T18:15:38.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 1 -- 172.21.15.84:0/4001956005 >> 172.21.15.84:0/4001956005 conn(0x7f240c0617a0 msgr2=0x7f240c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 1 -- 172.21.15.84:0/4001956005 shutdown_connections 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.761+0000 7f2411b47640 1 -- 172.21.15.84:0/4001956005 wait complete. 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.762+0000 7f2411b47640 10 monclient: build_initial_monmap 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.762+0000 7f2411b47640 10 monclient: monmap: 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:38.764 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.762+0000 7f2411b47640 1 librados: starting msgr at 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.762+0000 7f2411b47640 1 librados: starting objecter 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.762+0000 7f2411b47640 1 Processor -- start 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.763+0000 7f2411b47640 1 -- start start 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.763+0000 7f2411b47640 1 librados: setting wanted keys 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.763+0000 7f2411b47640 1 librados: calling monclient init 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.763+0000 7f2411b47640 10 monclient: init 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.763+0000 7f2411b47640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-c con 0x7f240c0b1730 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-a con 0x7f240c0b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient(hunting): picked mon.noname-b con 0x7f240c14be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:38.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c144150 con 0x7f240c0b4d60 2024-03-25T18:15:38.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c145840 con 0x7f240c14be90 2024-03-25T18:15:38.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c143c30 con 0x7f240c0b1730 2024-03-25T18:15:38.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:38.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.764+0000 7f2411b47640 10 monclient(hunting): authenticate will time out at 7583.063129s 2024-03-25T18:15:38.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.772+0000 7f240affd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f240c0b4d60 0x7f240c1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:36188/0 (socket says 172.21.15.84:36188) 2024-03-25T18:15:38.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.772+0000 7f240affd640 1 -- 172.21.15.84:0/2173940190 learned_addr learned my addr 172.21.15.84:0/2173940190 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:38.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.781+0000 7f23f3fff640 10 client.?.objecter ms_handle_connect 0x7f240c0b4d60 2024-03-25T18:15:38.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.781+0000 7f23f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:38.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.783+0000 7f23f3fff640 10 client.?.objecter ms_handle_connect 0x7f240c0b1730 2024-03-25T18:15:38.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.783+0000 7f23f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.790+0000 7f23f3fff640 10 client.?.objecter ms_handle_connect 0x7f240c14be90 2024-03-25T18:15:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.790+0000 7f23f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.791+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150780006 0 0) 0x7f240c143c30 con 0x7f240c0b1730 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.791+0000 7f23f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.791+0000 7f23f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.791+0000 7f23f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.791+0000 7f23f3fff640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:15:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f8004eb0 con 0x7f240c0b1730 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639905175 0 0) 0x7f240c144150 con 0x7f240c0b4d60 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:15:38.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.792+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c143c30 con 0x7f240c0b4d60 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2307346436 0 0) 0x7f23f8004eb0 con 0x7f240c0b1730 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240c144150 con 0x7f240c0b1730 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f23fc001b00 con 0x7f240c0b1730 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): dump: 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.808 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684584138 0 0) 0x7f240c143c30 con 0x7f240c0b4d60 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23f8004eb0 con 0x7f240c0b4d60 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2400001930 con 0x7f240c0b4d60 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.806+0000 7f23f3fff640 10 monclient(hunting): dump: 2024-03-25T18:15:38.809 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:38.810 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031016153 0 0) 0x7f240c145840 con 0x7f240c14be90 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:15:38.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.810+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c143c30 con 0x7f240c14be90 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 347195350 0 0) 0x7f240c143c30 con 0x7f240c14be90 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240c145840 con 0x7f240c14be90 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2404001ad0 con 0x7f240c14be90 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.822+0000 7f23f3fff640 10 monclient(hunting): dump: 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.824 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:38.825 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:38.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.832+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3447967494 0 0) 0x7f23f8004eb0 con 0x7f240c0b4d60 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.832+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 >> v1:172.21.15.84:6790/0 conn(0x7f240c0b1730 legacy=0x7f240c12fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.832+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 >> v1:172.21.15.157:6789/0 conn(0x7f240c14be90 legacy=0x7f240c1e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 1 monclient: found mon.a 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c143900 con 0x7f240c0b4d60 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f23f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 10 monclient: _renew_subs 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c137990 con 0x7f240c0b4d60 2024-03-25T18:15:38.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 10 monclient: _renew_subs 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c1307e0 con 0x7f240c0b4d60 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.833+0000 7f2411b47640 1 librados: init done 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.834+0000 7f2411b47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.834+0000 7f2411b47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.834+0000 7f2411b47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:38.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:38.834+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c1401f0 con 0x7f240c0b4d60 2024-03-25T18:15:39.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.141+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2400003610 con 0x7f240c0b4d60 2024-03-25T18:15:39.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.141+0000 7f23f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:39.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f24000039e0 con 0x7f240c0b4d60 2024-03-25T18:15:39.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f23f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f23f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f23f3fff640 10 monclient: dump: 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f2409ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.142+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2400003c00 con 0x7f240c0b4d60 2024-03-25T18:15:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.143+0000 7f240b7fe640 10 monclient: get_auth_request con 0x7f23f806e140 auth_method 0 2024-03-25T18:15:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.143+0000 7f23f3fff640 10 client.4701.objecter ms_handle_connect 0x7f23f806e140 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f2400054750 con 0x7f240c0b4d60 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 10 client.4701.objecter ms_dispatch 0x7f240c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2400050260 con 0x7f240c0b4d60 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.147+0000 7f23f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:39.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.376+0000 7f2411b47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.376+0000 7f2411b47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.376+0000 7f2411b47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:15:39.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.376+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f240c137b20 con 0x7f240c0b4d60 2024-03-25T18:15:39.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.410+0000 7f23f3fff640 1 -- 172.21.15.84:0/2173940190 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f2400052190 con 0x7f240c0b4d60 2024-03-25T18:15:39.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.411+0000 7f23f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.411+0000 7f23f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:39.412 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:39.413 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 10 librados: watch_flush enter 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 10 librados: watch_flush exit 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 10 client.4701.objecter successfully canceled tick 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 >> v1:172.21.15.157:6800/1801259026 conn(0x7f23f806e140 legacy=0x7f23f8070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 10 monclient: shutdown 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 >> v1:172.21.15.84:6789/0 conn(0x7f240c0b4d60 legacy=0x7f240c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 shutdown_connections 2024-03-25T18:15:39.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 >> 172.21.15.84:0/2173940190 conn(0x7f240c0617a0 msgr2=0x7f240c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:39.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 shutdown_connections 2024-03-25T18:15:39.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 -- 172.21.15.84:0/2173940190 wait complete. 2024-03-25T18:15:39.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:39.413+0000 7f2411b47640 1 librados: shutdown 2024-03-25T18:15:39.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:46.233 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.346+0000 7f9c18b2b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.346+0000 7f9c18b2b640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.346+0000 7f9c18b2b640 10 monclient: monmap: 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:46.347320+0000 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:46.347320+0000 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.347 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.348+0000 7f9c18b2b640 1 Processor -- start 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.348+0000 7f9c18b2b640 1 -- start start 2024-03-25T18:15:46.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.348+0000 7f9c18b2b640 10 monclient: init 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1406c380 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c140b12d0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9c140aedf0 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:46.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140b16f0 con 0x7f9c1406c380 2024-03-25T18:15:46.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140adfe0 con 0x7f9c140b12d0 2024-03-25T18:15:46.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140add00 con 0x7f9c140aedf0 2024-03-25T18:15:46.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.349+0000 7f9c18b2b640 10 monclient(hunting): authenticate will time out at 7590.647252s 2024-03-25T18:15:46.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.354+0000 7f9c03fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9c140b12d0 0x7f9c140ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51676/0 (socket says 172.21.15.84:51676) 2024-03-25T18:15:46.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.354+0000 7f9c03fff640 1 -- 172.21.15.84:0/2097539714 learned_addr learned my addr 172.21.15.84:0/2097539714 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:46.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098823656 0 0) 0x7f9c140b16f0 con 0x7f9c1406c380 2024-03-25T18:15:46.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc003520 con 0x7f9c1406c380 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169258334 0 0) 0x7f9c140adfe0 con 0x7f9c140b12d0 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:15:46.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.369+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c140b16f0 con 0x7f9c140b12d0 2024-03-25T18:15:46.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055492325 0 0) 0x7f9c140add00 con 0x7f9c140aedf0 2024-03-25T18:15:46.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:15:46.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.384+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c140adfe0 con 0x7f9c140aedf0 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127562544 0 0) 0x7f9c140b16f0 con 0x7f9c140b12d0 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 >> v1:172.21.15.84:6790/0 conn(0x7f9c140aedf0 legacy=0x7f9c1414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 >> v1:172.21.15.84:6789/0 conn(0x7f9c1406c380 legacy=0x7f9c140b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:15:46.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140ae2c0 con 0x7f9c140b12d0 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c080015f0 con 0x7f9c140b12d0 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.385+0000 7f9c11d9b640 10 monclient: dump: 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.387 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.388 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.386+0000 7f9c18b2b640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:15:46.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.386+0000 7f9c18b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c08001750 con 0x7f9c140b12d0 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 1 -- 172.21.15.84:0/2097539714 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c08001ae0 con 0x7f9c140b12d0 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c11d9b640 10 monclient: dump: 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.404 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c18b2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:46.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.403+0000 7f9c18b2b640 4 set_mon_vals no callback set 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 10 monclient: shutdown 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 1 -- 172.21.15.84:0/2097539714 >> v1:172.21.15.157:6789/0 conn(0x7f9c140b12d0 legacy=0x7f9c140ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 1 -- 172.21.15.84:0/2097539714 shutdown_connections 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 1 -- 172.21.15.84:0/2097539714 >> 172.21.15.84:0/2097539714 conn(0x7f9c140b5d10 msgr2=0x7f9c140b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 1 -- 172.21.15.84:0/2097539714 shutdown_connections 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.404+0000 7f9c18b2b640 1 -- 172.21.15.84:0/2097539714 wait complete. 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.405+0000 7f9c18b2b640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.405+0000 7f9c18b2b640 10 monclient: monmap: 2024-03-25T18:15:46.407 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.405+0000 7f9c18b2b640 1 librados: starting msgr at 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.405+0000 7f9c18b2b640 1 librados: starting objecter 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.405+0000 7f9c18b2b640 1 Processor -- start 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 -- start start 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 librados: setting wanted keys 2024-03-25T18:15:46.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 librados: calling monclient init 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient: init 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1406c380 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9c140aedf0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c140b12d0 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1406c8e0 con 0x7f9c1406c380 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141db590 con 0x7f9c140aedf0 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141db0b0 con 0x7f9c140b12d0 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.406+0000 7f9c18b2b640 10 monclient(hunting): authenticate will time out at 7590.704565s 2024-03-25T18:15:46.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.420+0000 7f9c03fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9c140aedf0 0x7f9c141c3b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51730/0 (socket says 172.21.15.84:51730) 2024-03-25T18:15:46.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.420+0000 7f9c03fff640 1 -- 172.21.15.84:0/1890249285 learned_addr learned my addr 172.21.15.84:0/1890249285 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:46.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.422+0000 7f9c02ffd640 10 client.?.objecter ms_handle_connect 0x7f9c140aedf0 2024-03-25T18:15:46.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.422+0000 7f9c02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.427+0000 7f9c02ffd640 10 client.?.objecter ms_handle_connect 0x7f9c1406c380 2024-03-25T18:15:46.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.427+0000 7f9c02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.431+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544811932 0 0) 0x7f9c141db590 con 0x7f9c140aedf0 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.432+0000 7f9c02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.432+0000 7f9c02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.432+0000 7f9c02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.432+0000 7f9c02ffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:15:46.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.432+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4004eb0 con 0x7f9c140aedf0 2024-03-25T18:15:46.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.435+0000 7f9c02ffd640 10 client.?.objecter ms_handle_connect 0x7f9c140b12d0 2024-03-25T18:15:46.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.435+0000 7f9c02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221701708 0 0) 0x7f9c141db0b0 con 0x7f9c140b12d0 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.445+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c141db590 con 0x7f9c140b12d0 2024-03-25T18:15:46.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705169535 0 0) 0x7f9c1406c8e0 con 0x7f9c1406c380 2024-03-25T18:15:46.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:15:46.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.446+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c141db0b0 con 0x7f9c1406c380 2024-03-25T18:15:46.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3049770146 0 0) 0x7f9bf4004eb0 con 0x7f9c140aedf0 2024-03-25T18:15:46.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c1406c8e0 con 0x7f9c140aedf0 2024-03-25T18:15:46.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c08001b20 con 0x7f9c140aedf0 2024-03-25T18:15:46.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 830598027 0 0) 0x7f9c141db590 con 0x7f9c140b12d0 2024-03-25T18:15:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf4004eb0 con 0x7f9c140b12d0 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c0c001870 con 0x7f9c140b12d0 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.453+0000 7f9c02ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.456 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181146708 0 0) 0x7f9c141db0b0 con 0x7f9c1406c380 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c141db590 con 0x7f9c1406c380 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c04001cd0 con 0x7f9c1406c380 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.460+0000 7f9c02ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:46.460 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.461 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 629519496 0 0) 0x7f9c1406c8e0 con 0x7f9c140aedf0 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 >> v1:172.21.15.84:6790/0 conn(0x7f9c140b12d0 legacy=0x7f9c141a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 >> v1:172.21.15.84:6789/0 conn(0x7f9c1406c380 legacy=0x7f9c141da9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 1 monclient: found mon.b 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c141db240 con 0x7f9c140aedf0 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 10 monclient: _renew_subs 2024-03-25T18:15:46.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c141c4870 con 0x7f9c140aedf0 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 10 monclient: _renew_subs 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c141c4270 con 0x7f9c140aedf0 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.464+0000 7f9c18b2b640 1 librados: init done 2024-03-25T18:15:46.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.465+0000 7f9c18b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.465+0000 7f9c18b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.465+0000 7f9c18b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:46.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.465+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c140627b0 con 0x7f9c140aedf0 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c08004a90 con 0x7f9c140aedf0 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f9c080011b0 con 0x7f9c140aedf0 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c02ffd640 10 monclient: dump: 2024-03-25T18:15:46.736 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.735+0000 7f9c1159a640 4 set_mon_vals no callback set 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.736+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f9c0801ac10 con 0x7f9c140aedf0 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.737+0000 7f9c1259c640 10 monclient: get_auth_request con 0x7f9bf406e0f0 auth_method 0 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.737+0000 7f9c02ffd640 10 client.4705.objecter ms_handle_connect 0x7f9bf406e0f0 2024-03-25T18:15:46.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.739+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f9c08054830 con 0x7f9c140aedf0 2024-03-25T18:15:46.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.739+0000 7f9c02ffd640 10 client.4705.objecter ms_dispatch 0x7f9c14000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:46.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.739+0000 7f9c02ffd640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:46.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.739+0000 7f9c02ffd640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:46.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.739+0000 7f9c02ffd640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:15:46.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.741+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c08050170 con 0x7f9c140aedf0 2024-03-25T18:15:46.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.741+0000 7f9c02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:46.741+0000 7f9c02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:47.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.000+0000 7f9c18b2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.000+0000 7f9c18b2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.000+0000 7f9c18b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:15:47.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.000+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c14069f40 con 0x7f9c140aedf0 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.022+0000 7f9c02ffd640 1 -- 172.21.15.84:0/1890249285 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f9c08001d40 con 0x7f9c140aedf0 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.022+0000 7f9c02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.022+0000 7f9c02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 10 librados: watch_flush enter 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 10 librados: watch_flush exit 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 >> v1:172.21.15.157:6800/1801259026 conn(0x7f9bf406e0f0 legacy=0x7f9bf40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 10 monclient: shutdown 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.024+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 >> v1:172.21.15.157:6789/0 conn(0x7f9c140aedf0 legacy=0x7f9c141c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.025+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 shutdown_connections 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.025+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 >> 172.21.15.84:0/1890249285 conn(0x7f9c140b5d10 msgr2=0x7f9c141c9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.025+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 shutdown_connections 2024-03-25T18:15:47.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.025+0000 7f9c18b2b640 1 -- 172.21.15.84:0/1890249285 wait complete. 2024-03-25T18:15:47.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:47.025+0000 7f9c18b2b640 1 librados: shutdown 2024-03-25T18:15:47.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:49.342 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:46.160 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.419 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:46.238 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.453 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:46.271 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.456 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:46.239 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.458 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:46.277 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.459 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:46.278 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.462 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:46.245 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.482 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:10:46.266 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.485 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:10:46.304 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.843 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.953+0000 7fb658463640 10 monclient: get_monmap_and_config 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.953+0000 7fb658463640 10 monclient: build_initial_monmap 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.953+0000 7fb658463640 10 monclient: monmap: 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:15:53.953964+0000 2024-03-25T18:15:53.954 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:15:53.953964+0000 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:15:53.955 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.954+0000 7fb658463640 1 Processor -- start 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.954+0000 7fb658463640 1 -- start start 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.954+0000 7fb658463640 10 monclient: init 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient(hunting): picked mon.noname-c con 0x7fb6500a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:53.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient(hunting): picked mon.noname-a con 0x7fb65013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient(hunting): picked mon.noname-b con 0x7fb6501430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb650142520 con 0x7fb65013f710 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6500b28a0 con 0x7fb6500a98f0 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6500b2a30 con 0x7fb6501430c0 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:53.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.955+0000 7fb658463640 10 monclient(hunting): authenticate will time out at 7598.253541s 2024-03-25T18:15:53.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.962+0000 7fb656a00640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb6501430c0 0x7fb65014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57476/0 (socket says 172.21.15.84:57476) 2024-03-25T18:15:53.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.962+0000 7fb656a00640 1 -- 172.21.15.84:0/1067376475 learned_addr learned my addr 172.21.15.84:0/1067376475 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:53.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424235240 0 0) 0x7fb650142520 con 0x7fb65013f710 2024-03-25T18:15:53.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.975+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb63c003520 con 0x7fb65013f710 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434968904 0 0) 0x7fb6500b2a30 con 0x7fb6501430c0 2024-03-25T18:15:53.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:53.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:53.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:53.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:15:53.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.976+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb650142520 con 0x7fb6501430c0 2024-03-25T18:15:53.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.983+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3015780271 0 0) 0x7fb63c003520 con 0x7fb65013f710 2024-03-25T18:15:53.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 >> v1:172.21.15.84:6790/0 conn(0x7fb6501430c0 legacy=0x7fb65014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:53.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 >> v1:172.21.15.157:6789/0 conn(0x7fb6500a98f0 legacy=0x7fb6500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:53.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 1 monclient: found mon.noname-a 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500b44c0 con 0x7fb65013f710 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: _finish_auth 0 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb640001690 con 0x7fb65013f710 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb658463640 5 monclient: authenticate success, global_id 4710 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb658463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:53.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb6551fd640 10 monclient: dump: 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:53.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.984+0000 7fb658463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6400017f0 con 0x7fb65013f710 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 1 -- 172.21.15.84:0/1067376475 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb640001c00 con 0x7fb65013f710 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb658463640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb658463640 4 set_mon_vals no callback set 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.990+0000 7fb6551fd640 10 monclient: dump: 2024-03-25T18:15:53.991 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:53.992 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:53.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 10 monclient: shutdown 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 1 -- 172.21.15.84:0/1067376475 >> v1:172.21.15.84:6789/0 conn(0x7fb65013f710 legacy=0x7fb6501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 1 -- 172.21.15.84:0/1067376475 shutdown_connections 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 1 -- 172.21.15.84:0/1067376475 >> 172.21.15.84:0/1067376475 conn(0x7fb6500b7ea0 msgr2=0x7fb6500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 1 -- 172.21.15.84:0/1067376475 shutdown_connections 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.991+0000 7fb658463640 1 -- 172.21.15.84:0/1067376475 wait complete. 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.992+0000 7fb658463640 10 monclient: build_initial_monmap 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.992+0000 7fb658463640 10 monclient: monmap: 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:15:53.994 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 librados: starting msgr at 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 librados: starting objecter 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 Processor -- start 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 -- start start 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 librados: setting wanted keys 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 1 librados: calling monclient init 2024-03-25T18:15:53.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.993+0000 7fb658463640 10 monclient: init 2024-03-25T18:15:53.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient(hunting): picked mon.noname-b con 0x7fb6500a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient(hunting): picked mon.noname-c con 0x7fb65013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient(hunting): picked mon.noname-a con 0x7fb6501430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65014c400 con 0x7fb6501430c0 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65014c000 con 0x7fb6500a98f0 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65014c770 con 0x7fb65013f710 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:53.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:53.994+0000 7fb658463640 10 monclient(hunting): authenticate will time out at 7598.292464s 2024-03-25T18:15:54.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.011+0000 7fb6559fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb65013f710 0x7fb6501db900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57508/0 (socket says 172.21.15.84:57508) 2024-03-25T18:15:54.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.011+0000 7fb6559fe640 1 -- 172.21.15.84:0/1160003036 learned_addr learned my addr 172.21.15.84:0/1160003036 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:15:54.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.013+0000 7fb646ffd640 10 client.?.objecter ms_handle_connect 0x7fb65013f710 2024-03-25T18:15:54.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.013+0000 7fb646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.016+0000 7fb646ffd640 10 client.?.objecter ms_handle_connect 0x7fb6501430c0 2024-03-25T18:15:54.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.016+0000 7fb646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.019+0000 7fb646ffd640 10 client.?.objecter ms_handle_connect 0x7fb6500a98f0 2024-03-25T18:15:54.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.019+0000 7fb646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889734759 0 0) 0x7fb65014c770 con 0x7fb65013f710 2024-03-25T18:15:54.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:15:54.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.024+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634004eb0 con 0x7fb65013f710 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 533736593 0 0) 0x7fb634004eb0 con 0x7fb65013f710 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb65014c770 con 0x7fb65013f710 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb640001690 con 0x7fb65013f710 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:54.036 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:54.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.035+0000 7fb646ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718160236 0 0) 0x7fb65014c400 con 0x7fb6501430c0 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:15:54.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.036+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634004eb0 con 0x7fb6501430c0 2024-03-25T18:15:54.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530014782 0 0) 0x7fb65014c770 con 0x7fb65013f710 2024-03-25T18:15:54.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 >> v1:172.21.15.157:6789/0 conn(0x7fb6500a98f0 legacy=0x7fb650134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 >> v1:172.21.15.84:6789/0 conn(0x7fb6501430c0 legacy=0x7fb6501dc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:54.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 1 monclient: found mon.c 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65014c900 con 0x7fb65013f710 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 10 monclient: _renew_subs 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6500b7a20 con 0x7fb65013f710 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 10 monclient: _renew_subs 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:54.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6501358a0 con 0x7fb65013f710 2024-03-25T18:15:54.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.046+0000 7fb658463640 1 librados: init done 2024-03-25T18:15:54.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.047+0000 7fb658463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.047+0000 7fb658463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.047+0000 7fb658463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:54.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.047+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65014c190 con 0x7fb65013f710 2024-03-25T18:15:54.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6400038a0 con 0x7fb65013f710 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb640003cf0 con 0x7fb65013f710 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb646ffd640 10 monclient: dump: 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:15:54.344 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:15:54.345 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:15:54.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.342+0000 7fb6549fc640 4 set_mon_vals no callback set 2024-03-25T18:15:54.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.344+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fb640003f10 con 0x7fb65013f710 2024-03-25T18:15:54.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.344+0000 7fb6561ff640 10 monclient: get_auth_request con 0x7fb63406dd90 auth_method 0 2024-03-25T18:15:54.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.344+0000 7fb646ffd640 10 client.4703.objecter ms_handle_connect 0x7fb63406dd90 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.356+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7fb640054900 con 0x7fb65013f710 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.356+0000 7fb646ffd640 10 client.4703.objecter ms_dispatch 0x7fb650000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.356+0000 7fb646ffd640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.356+0000 7fb646ffd640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.357+0000 7fb646ffd640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T18:15:54.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.357+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb640050410 con 0x7fb65013f710 2024-03-25T18:15:54.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.357+0000 7fb646ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.357+0000 7fb646ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:54.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.606+0000 7fb658463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:54.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.606+0000 7fb658463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:54.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.606+0000 7fb658463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:15:54.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.606+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb65014c5b0 con 0x7fb65013f710 2024-03-25T18:15:54.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.673+0000 7fb646ffd640 1 -- 172.21.15.84:0/1160003036 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7fb65014c5b0 con 0x7fb65013f710 2024-03-25T18:15:54.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.673+0000 7fb646ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:54.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.673+0000 7fb646ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:54.675 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:15:54.675 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:54.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.675+0000 7fb658463640 10 librados: watch_flush enter 2024-03-25T18:15:54.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 10 librados: watch_flush exit 2024-03-25T18:15:54.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T18:15:54.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 10 client.4703.objecter successfully canceled tick 2024-03-25T18:15:54.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 >> v1:172.21.15.157:6800/1801259026 conn(0x7fb63406dd90 legacy=0x7fb634070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 10 monclient: shutdown 2024-03-25T18:15:54.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 >> v1:172.21.15.84:6790/0 conn(0x7fb65013f710 legacy=0x7fb6501db900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 shutdown_connections 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 >> 172.21.15.84:0/1160003036 conn(0x7fb6500b7ea0 msgr2=0x7fb650074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 shutdown_connections 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 -- 172.21.15.84:0/1160003036 wait complete. 2024-03-25T18:15:54.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:15:54.676+0000 7fb658463640 1 librados: shutdown 2024-03-25T18:15:54.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:01.496 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.607+0000 7f00b693d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.607+0000 7f00b693d640 10 monclient: build_initial_monmap 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.607+0000 7f00b693d640 10 monclient: monmap: 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:01.608075+0000 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:01.608075+0000 2024-03-25T18:16:01.608 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:01.609 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:01.609 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:01.609 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:01.609 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:01.609 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.608+0000 7f00b693d640 1 Processor -- start 2024-03-25T18:16:01.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.608+0000 7f00b693d640 1 -- start start 2024-03-25T18:16:01.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.608+0000 7f00b693d640 10 monclient: init 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-b con 0x7f00b00b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-a con 0x7f00b00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-c con 0x7f00b014be90 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00af940 con 0x7f00b00b1730 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00b0a70 con 0x7f00b014be90 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00b0d50 con 0x7f00b00b4d60 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:01.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.609+0000 7f00b693d640 10 monclient(hunting): authenticate will time out at 7605.907260s 2024-03-25T18:16:01.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.618+0000 7f00af7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f00b00b1730 0x7f00b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:50202/0 (socket says 172.21.15.84:50202) 2024-03-25T18:16:01.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.618+0000 7f00af7fe640 1 -- 172.21.15.84:0/2699391973 learned_addr learned my addr 172.21.15.84:0/2699391973 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:01.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279624554 0 0) 0x7f00b00af940 con 0x7f00b00b1730 2024-03-25T18:16:01.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:16:01.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.629+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098003520 con 0x7f00b00b1730 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351607742 0 0) 0x7f00b00b0d50 con 0x7f00b00b4d60 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:16:01.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.632+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b00af940 con 0x7f00b00b4d60 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2657738181 0 0) 0x7f0098003520 con 0x7f00b00b1730 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 >> v1:172.21.15.84:6790/0 conn(0x7f00b00b4d60 legacy=0x7f00b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 >> v1:172.21.15.157:6789/0 conn(0x7f00b014be90 legacy=0x7f00b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 1 monclient: found mon.noname-a 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:16:01.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.633+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00b1030 con 0x7f00b00b1730 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f00a4001690 con 0x7f00b00b1730 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00aeffd640 10 monclient: dump: 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:01.635 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:01.636 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:01.636 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:01.636 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00b693d640 5 monclient: authenticate success, global_id 4716 2024-03-25T18:16:01.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.634+0000 7f00b693d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:01.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a40017f0 con 0x7f00b00b1730 2024-03-25T18:16:01.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:01.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00b693d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:01.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00b693d640 4 set_mon_vals no callback set 2024-03-25T18:16:01.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 1 -- 172.21.15.84:0/2699391973 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f00a4001c00 con 0x7f00b00b1730 2024-03-25T18:16:01.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.637+0000 7f00aeffd640 10 monclient: dump: 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 10 monclient: shutdown 2024-03-25T18:16:01.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 1 -- 172.21.15.84:0/2699391973 >> v1:172.21.15.84:6789/0 conn(0x7f00b00b1730 legacy=0x7f00b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 1 -- 172.21.15.84:0/2699391973 shutdown_connections 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 1 -- 172.21.15.84:0/2699391973 >> 172.21.15.84:0/2699391973 conn(0x7f00b00617a0 msgr2=0x7f00b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 1 -- 172.21.15.84:0/2699391973 shutdown_connections 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 1 -- 172.21.15.84:0/2699391973 wait complete. 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 10 monclient: build_initial_monmap 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.638+0000 7f00b693d640 10 monclient: monmap: 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:01.641 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 librados: starting msgr at 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 librados: starting objecter 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 Processor -- start 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 -- start start 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 librados: setting wanted keys 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 1 librados: calling monclient init 2024-03-25T18:16:01.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.639+0000 7f00b693d640 10 monclient: init 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-a con 0x7f00b00b1730 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-b con 0x7f00b00b4d60 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient(hunting): picked mon.noname-c con 0x7f00b014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b0140e50 con 0x7f00b00b1730 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b0140930 con 0x7f00b00b4d60 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b0138760 con 0x7f00b014be90 2024-03-25T18:16:01.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:01.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.640+0000 7f00b693d640 10 monclient(hunting): authenticate will time out at 7605.938227s 2024-03-25T18:16:01.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.653+0000 7f00b4eda640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f00b014be90 0x7f00b01e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56364/0 (socket says 172.21.15.84:56364) 2024-03-25T18:16:01.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.653+0000 7f00b4eda640 1 -- 172.21.15.84:0/4199875662 learned_addr learned my addr 172.21.15.84:0/4199875662 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:01.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.654+0000 7f00acff9640 10 client.?.objecter ms_handle_connect 0x7f00b00b4d60 2024-03-25T18:16:01.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.654+0000 7f00acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:01.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.654+0000 7f00acff9640 10 client.?.objecter ms_handle_connect 0x7f00b014be90 2024-03-25T18:16:01.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.654+0000 7f00acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:01.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.662+0000 7f00acff9640 10 client.?.objecter ms_handle_connect 0x7f00b00b1730 2024-03-25T18:16:01.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.662+0000 7f00acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416441785 0 0) 0x7f00b0138760 con 0x7f00b014be90 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:16:01.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.669+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c004eb0 con 0x7f00b014be90 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442681921 0 0) 0x7f00b0140930 con 0x7f00b00b4d60 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:16:01.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.670+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b0138760 con 0x7f00b00b4d60 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.688+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3288032578 0 0) 0x7f009c004eb0 con 0x7f00b014be90 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00b0140930 con 0x7f00b014be90 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f00a8001c30 con 0x7f00b014be90 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): dump: 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.690 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868915677 0 0) 0x7f00b0140e50 con 0x7f00b00b1730 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c004eb0 con 0x7f00b00b1730 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644605873 0 0) 0x7f00b0138760 con 0x7f00b00b4d60 2024-03-25T18:16:01.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00b0140e50 con 0x7f00b00b4d60 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f00a4001b00 con 0x7f00b00b4d60 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.689+0000 7f00acff9640 10 monclient(hunting): dump: 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:01.692 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:01.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.706+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3560744244 0 0) 0x7f00b0140e50 con 0x7f00b00b4d60 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 >> v1:172.21.15.84:6790/0 conn(0x7f00b014be90 legacy=0x7f00b01e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 >> v1:172.21.15.84:6789/0 conn(0x7f00b00b1730 legacy=0x7f00b0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 1 monclient: found mon.b 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b01388f0 con 0x7f00b00b4d60 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:16:01.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 10 monclient: _renew_subs 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b0137e90 con 0x7f00b00b4d60 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 10 monclient: _renew_subs 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b01384a0 con 0x7f00b00b4d60 2024-03-25T18:16:01.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.707+0000 7f00b693d640 1 librados: init done 2024-03-25T18:16:01.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.708+0000 7f00b693d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:01.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.708+0000 7f00b693d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:01.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.708+0000 7f00b693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:01.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:01.708+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b0140ac0 con 0x7f00b00b4d60 2024-03-25T18:16:02.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a4001c60 con 0x7f00b00b4d60 2024-03-25T18:16:02.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:02.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f00a4004c30 con 0x7f00b00b4d60 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00acff9640 10 monclient: dump: 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:02.008 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.006+0000 7f00ae7fc640 4 set_mon_vals no callback set 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.007+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f00a401ac00 con 0x7f00b00b4d60 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.008+0000 7f00affff640 10 monclient: get_auth_request con 0x7f009c06df00 auth_method 0 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.008+0000 7f00acff9640 10 client.4717.objecter ms_handle_connect 0x7f009c06df00 2024-03-25T18:16:02.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.012+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f00a40547b0 con 0x7f00b00b4d60 2024-03-25T18:16:02.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.012+0000 7f00acff9640 10 client.4717.objecter ms_dispatch 0x7f00b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:02.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.012+0000 7f00acff9640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:02.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.012+0000 7f00acff9640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:02.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.012+0000 7f00acff9640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:16:02.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.013+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00a40502c0 con 0x7f00b00b4d60 2024-03-25T18:16:02.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.013+0000 7f00acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.013+0000 7f00acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.237+0000 7f00b693d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.237+0000 7f00b693d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.237+0000 7f00b693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.237+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00b01401c0 con 0x7f00b00b4d60 2024-03-25T18:16:02.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.278+0000 7f00acff9640 1 -- 172.21.15.84:0/4199875662 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f00a40521f0 con 0x7f00b00b4d60 2024-03-25T18:16:02.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.278+0000 7f00acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.278+0000 7f00acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:02.279 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:02.279 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:02.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 10 librados: watch_flush enter 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 10 librados: watch_flush exit 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 >> v1:172.21.15.157:6800/1801259026 conn(0x7f009c06df00 legacy=0x7f009c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 10 monclient: shutdown 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 >> v1:172.21.15.157:6789/0 conn(0x7f00b00b4d60 legacy=0x7f00b01dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 shutdown_connections 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 >> 172.21.15.84:0/4199875662 conn(0x7f00b00617a0 msgr2=0x7f00b013c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 shutdown_connections 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 -- 172.21.15.84:0/4199875662 wait complete. 2024-03-25T18:16:02.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:02.280+0000 7f00b693d640 1 librados: shutdown 2024-03-25T18:16:02.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:09.098 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.213+0000 7f29f88c9640 10 monclient: get_monmap_and_config 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.213+0000 7f29f88c9640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.213+0000 7f29f88c9640 10 monclient: monmap: 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:09.215296+0000 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:09.215296+0000 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:09.215 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:09.216 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:09.216 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.214+0000 7f29f88c9640 1 Processor -- start 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.214+0000 7f29f88c9640 1 -- start start 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.214+0000 7f29f88c9640 10 monclient: init 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-b con 0x7f29f40a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-c con 0x7f29f413f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-a con 0x7f29f41430c0 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:09.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f4142520 con 0x7f29f41430c0 2024-03-25T18:16:09.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40b28a0 con 0x7f29f413f710 2024-03-25T18:16:09.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40b2a30 con 0x7f29f40a98f0 2024-03-25T18:16:09.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.215+0000 7f29f88c9640 10 monclient(hunting): authenticate will time out at 7613.514446s 2024-03-25T18:16:09.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.220+0000 7f29ebfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f29f413f710 0x7f29f41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:51282/0 (socket says 172.21.15.84:51282) 2024-03-25T18:16:09.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.220+0000 7f29ebfff640 1 -- 172.21.15.84:0/3338762434 learned_addr learned my addr 172.21.15.84:0/3338762434 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:09.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41344858 0 0) 0x7f29f40b28a0 con 0x7f29f413f710 2024-03-25T18:16:09.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:16:09.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.226+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e0003520 con 0x7f29f413f710 2024-03-25T18:16:09.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798133397 0 0) 0x7f29f40b2a30 con 0x7f29f40a98f0 2024-03-25T18:16:09.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:16:09.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.233+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f40b28a0 con 0x7f29f40a98f0 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131742891 0 0) 0x7f29e0003520 con 0x7f29f413f710 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 >> v1:172.21.15.84:6790/0 conn(0x7f29f40a98f0 legacy=0x7f29f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 >> v1:172.21.15.84:6789/0 conn(0x7f29f41430c0 legacy=0x7f29f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40b44c0 con 0x7f29f413f710 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f29dc001690 con 0x7f29f413f710 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f1d9b640 10 monclient: dump: 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f88c9640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:16:09.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.236+0000 7f29f88c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29dc0017f0 con 0x7f29f413f710 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 1 -- 172.21.15.84:0/3338762434 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f29dc001c00 con 0x7f29f413f710 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f1d9b640 10 monclient: dump: 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:09.243 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f88c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.241+0000 7f29f88c9640 4 set_mon_vals no callback set 2024-03-25T18:16:09.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.242+0000 7f29f88c9640 10 monclient: shutdown 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.242+0000 7f29f88c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.242+0000 7f29f88c9640 1 -- 172.21.15.84:0/3338762434 >> v1:172.21.15.157:6789/0 conn(0x7f29f413f710 legacy=0x7f29f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.242+0000 7f29f88c9640 1 -- 172.21.15.84:0/3338762434 shutdown_connections 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.242+0000 7f29f88c9640 1 -- 172.21.15.84:0/3338762434 >> 172.21.15.84:0/3338762434 conn(0x7f29f40b7ea0 msgr2=0x7f29f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.243+0000 7f29f88c9640 1 -- 172.21.15.84:0/3338762434 shutdown_connections 2024-03-25T18:16:09.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.243+0000 7f29f88c9640 1 -- 172.21.15.84:0/3338762434 wait complete. 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 10 monclient: monmap: 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:09.246 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 librados: starting msgr at 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 librados: starting objecter 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 Processor -- start 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 -- start start 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 librados: setting wanted keys 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 1 librados: calling monclient init 2024-03-25T18:16:09.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.244+0000 7f29f88c9640 10 monclient: init 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-b con 0x7f29f40a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-a con 0x7f29f413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient(hunting): picked mon.noname-c con 0x7f29f41430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40ac010 con 0x7f29f413f710 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40ac2f0 con 0x7f29f40a98f0 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40ac5d0 con 0x7f29f41430c0 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.245+0000 7f29f88c9640 10 monclient(hunting): authenticate will time out at 7613.544560s 2024-03-25T18:16:09.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.251+0000 7f29ebfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f29f413f710 0x7f29f4070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47186/0 (socket says 172.21.15.84:47186) 2024-03-25T18:16:09.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.251+0000 7f29ebfff640 1 -- 172.21.15.84:0/2880126747 learned_addr learned my addr 172.21.15.84:0/2880126747 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:09.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.255+0000 7f29eaffd640 10 client.?.objecter ms_handle_connect 0x7f29f413f710 2024-03-25T18:16:09.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.255+0000 7f29eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.261+0000 7f29eaffd640 10 client.?.objecter ms_handle_connect 0x7f29f41430c0 2024-03-25T18:16:09.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.261+0000 7f29eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304551821 0 0) 0x7f29f40ac010 con 0x7f29f413f710 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:16:09.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.269+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d4004eb0 con 0x7f29f413f710 2024-03-25T18:16:09.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.280+0000 7f29eaffd640 10 client.?.objecter ms_handle_connect 0x7f29f40a98f0 2024-03-25T18:16:09.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.280+0000 7f29eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248217197 0 0) 0x7f29f40ac5d0 con 0x7f29f41430c0 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f40ac010 con 0x7f29f41430c0 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877968074 0 0) 0x7f29d4004eb0 con 0x7f29f413f710 2024-03-25T18:16:09.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29f40ac5d0 con 0x7f29f413f710 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f29dc001bc0 con 0x7f29f413f710 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): dump: 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:09.284 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:09.285 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:09.285 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.281+0000 7f29eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:09.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776578258 0 0) 0x7f29f40ac5d0 con 0x7f29f413f710 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 >> v1:172.21.15.84:6790/0 conn(0x7f29f41430c0 legacy=0x7f29f40b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 >> v1:172.21.15.157:6789/0 conn(0x7f29f40a98f0 legacy=0x7f29f4071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 1 monclient: found mon.a 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f406f110 con 0x7f29f413f710 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:16:09.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 10 monclient: _renew_subs 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f40b0510 con 0x7f29f413f710 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 10 client.4722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 10 monclient: _renew_subs 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f41e7c50 con 0x7f29f413f710 2024-03-25T18:16:09.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.287+0000 7f29f88c9640 1 librados: init done 2024-03-25T18:16:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.288+0000 7f29f88c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.289+0000 7f29f88c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.289+0000 7f29f88c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.289+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f40627b0 con 0x7f29f413f710 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29dc0011f0 con 0x7f29f413f710 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f29dc004cb0 con 0x7f29f413f710 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29eaffd640 10 monclient: dump: 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.584 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.582+0000 7f29f159a640 4 set_mon_vals no callback set 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.583+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f29dc01acc0 con 0x7f29f413f710 2024-03-25T18:16:09.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.583+0000 7f29f259c640 10 monclient: get_auth_request con 0x7f29d406dc20 auth_method 0 2024-03-25T18:16:09.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.584+0000 7f29eaffd640 10 client.4722.objecter ms_handle_connect 0x7f29d406dc20 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f29dc0549c0 con 0x7f29f413f710 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 10 client.4722.objecter ms_dispatch 0x7f29f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 3 client.4722.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 3 client.4722.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 20 client.4722.objecter dump_active .. 0 homeless 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29dc050300 con 0x7f29f413f710 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.589+0000 7f29eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:09.612 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:06.431 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.659 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:06.443 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.660 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:06.444 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.663 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:06.482 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.671 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:06.455 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.688 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:06.507 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.753 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:06.571 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.764 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:06.583 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.784 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:06.604 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:09.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.789+0000 7f29f88c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:09.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.789+0000 7f29f88c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:09.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.789+0000 7f29f88c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:09.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.789+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29f406f660 con 0x7f29f413f710 2024-03-25T18:16:09.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.964+0000 7f29eaffd640 1 -- 172.21.15.84:0/2880126747 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f29dc052400 con 0x7f29f413f710 2024-03-25T18:16:09.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.964+0000 7f29eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:09.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.964+0000 7f29eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:09.966 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:09.966 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.966+0000 7f29f88c9640 10 librados: watch_flush enter 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.966+0000 7f29f88c9640 10 librados: watch_flush exit 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.966+0000 7f29f88c9640 20 client.4722.objecter shutdown clearing up homeless session... 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.966+0000 7f29f88c9640 10 client.4722.objecter successfully canceled tick 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 >> v1:172.21.15.157:6800/1801259026 conn(0x7f29d406dc20 legacy=0x7f29d40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 10 monclient: shutdown 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 >> v1:172.21.15.84:6789/0 conn(0x7f29f413f710 legacy=0x7f29f4070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 shutdown_connections 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 >> 172.21.15.84:0/2880126747 conn(0x7f29f40b7ea0 msgr2=0x7f29f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 shutdown_connections 2024-03-25T18:16:09.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 -- 172.21.15.84:0/2880126747 wait complete. 2024-03-25T18:16:09.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:09.967+0000 7f29f88c9640 1 librados: shutdown 2024-03-25T18:16:09.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:16.787 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 10 monclient: get_monmap_and_config 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 10 monclient: build_initial_monmap 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 10 monclient: monmap: 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:16.901446+0000 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:16.901446+0000 2024-03-25T18:16:16.901 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 1 Processor -- start 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 1 -- start start 2024-03-25T18:16:16.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.900+0000 7f1fe4e55640 10 monclient: init 2024-03-25T18:16:16.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:16.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:16.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe00b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00af940 con 0x7f1fe00b4d60 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00b0a70 con 0x7f1fe00b1730 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00b0d50 con 0x7f1fe014be90 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:16.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.901+0000 7f1fe4e55640 10 monclient(hunting): authenticate will time out at 7621.200850s 2024-03-25T18:16:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.905+0000 7f1fddd9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1fe00b1730 0x7f1fe00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56156/0 (socket says 172.21.15.84:56156) 2024-03-25T18:16:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.905+0000 7f1fddd9b640 1 -- 172.21.15.84:0/177216197 learned_addr learned my addr 172.21.15.84:0/177216197 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675242173 0 0) 0x7f1fe00b0a70 con 0x7f1fe00b1730 2024-03-25T18:16:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:16:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.914+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8003520 con 0x7f1fe00b1730 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3482361134 0 0) 0x7f1fc8003520 con 0x7f1fe00b1730 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 >> v1:172.21.15.84:6790/0 conn(0x7f1fe014be90 legacy=0x7f1fe014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 >> v1:172.21.15.84:6789/0 conn(0x7f1fe00b4d60 legacy=0x7f1fe013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 monclient: found mon.noname-c 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00b1030 con 0x7f1fe00b1730 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: _finish_auth 0 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1fd4001690 con 0x7f1fe00b1730 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:16.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fdd59a640 10 monclient: dump: 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fe4e55640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:16:16.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.931+0000 7f1fe4e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd40017f0 con 0x7f1fe00b1730 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 1 -- 172.21.15.84:0/177216197 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1fd4001c00 con 0x7f1fe00b1730 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fdd59a640 10 monclient: dump: 2024-03-25T18:16:16.940 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fe4e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:16.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.939+0000 7f1fe4e55640 4 set_mon_vals no callback set 2024-03-25T18:16:16.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 10 monclient: shutdown 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 1 -- 172.21.15.84:0/177216197 >> v1:172.21.15.157:6789/0 conn(0x7f1fe00b1730 legacy=0x7f1fe00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 1 -- 172.21.15.84:0/177216197 shutdown_connections 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 1 -- 172.21.15.84:0/177216197 >> 172.21.15.84:0/177216197 conn(0x7f1fe00617a0 msgr2=0x7f1fe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 1 -- 172.21.15.84:0/177216197 shutdown_connections 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.940+0000 7f1fe4e55640 1 -- 172.21.15.84:0/177216197 wait complete. 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 10 monclient: build_initial_monmap 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 10 monclient: monmap: 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:16.943 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 librados: starting msgr at 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 librados: starting objecter 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 Processor -- start 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 -- start start 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 librados: setting wanted keys 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 1 librados: calling monclient init 2024-03-25T18:16:16.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.941+0000 7f1fe4e55640 10 monclient: init 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe00b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00b4d60 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe014be90 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0145540 con 0x7f1fe00b4d60 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe013ac50 con 0x7f1fe00b1730 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe012f7b0 con 0x7f1fe014be90 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:16.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.942+0000 7f1fe4e55640 10 monclient(hunting): authenticate will time out at 7621.241507s 2024-03-25T18:16:16.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.959+0000 7f1fde59c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1fe00b1730 0x7f1fe0139190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:56174/0 (socket says 172.21.15.84:56174) 2024-03-25T18:16:16.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.959+0000 7f1fde59c640 1 -- 172.21.15.84:0/2311358695 learned_addr learned my addr 172.21.15.84:0/2311358695 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:16.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.961+0000 7f1fceffd640 10 client.?.objecter ms_handle_connect 0x7f1fe00b1730 2024-03-25T18:16:16.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.961+0000 7f1fceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:16.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.963+0000 7f1fceffd640 10 client.?.objecter ms_handle_connect 0x7f1fe014be90 2024-03-25T18:16:16.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.964+0000 7f1fceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:16.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.966+0000 7f1fceffd640 10 client.?.objecter ms_handle_connect 0x7f1fe00b4d60 2024-03-25T18:16:16.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.966+0000 7f1fceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185384678 0 0) 0x7f1fe012f7b0 con 0x7f1fe014be90 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:16:16.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.979+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0004eb0 con 0x7f1fe014be90 2024-03-25T18:16:16.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335054502 0 0) 0x7f1fe013ac50 con 0x7f1fe00b1730 2024-03-25T18:16:16.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:16.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:16.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:16.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:16:16.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.986+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe012f7b0 con 0x7f1fe00b1730 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331596036 0 0) 0x7f1fe0145540 con 0x7f1fe00b4d60 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:16:16.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.990+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe013ac50 con 0x7f1fe00b4d60 2024-03-25T18:16:16.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838677946 0 0) 0x7f1fc0004eb0 con 0x7f1fe014be90 2024-03-25T18:16:16.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe0145540 con 0x7f1fe014be90 2024-03-25T18:16:16.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1fd8001b90 con 0x7f1fe014be90 2024-03-25T18:16:16.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 10 monclient(hunting): dump: 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:16.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:16.997+0000 7f1fceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4225189990 0 0) 0x7f1fe0145540 con 0x7f1fe014be90 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 >> v1:172.21.15.157:6789/0 conn(0x7f1fe00b1730 legacy=0x7f1fe0139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 >> v1:172.21.15.84:6789/0 conn(0x7f1fe00b4d60 legacy=0x7f1fe012e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 1 monclient: found mon.c 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe013af30 con 0x7f1fe014be90 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 5 monclient: authenticate success, global_id 4721 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 10 monclient: _renew_subs 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe01451d0 con 0x7f1fe014be90 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 10 monclient: _renew_subs 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0139ca0 con 0x7f1fe014be90 2024-03-25T18:16:17.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.020+0000 7f1fe4e55640 1 librados: init done 2024-03-25T18:16:17.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.021+0000 7f1fe4e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.021+0000 7f1fe4e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.021+0000 7f1fe4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:16:17.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.021+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe01401d0 con 0x7f1fe014be90 2024-03-25T18:16:17.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8002080 con 0x7f1fe014be90 2024-03-25T18:16:17.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f1fd8003b30 con 0x7f1fe014be90 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fceffd640 10 monclient: dump: 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:17.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.356+0000 7f1fdcd99640 4 set_mon_vals no callback set 2024-03-25T18:16:17.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.357+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f1fd8003d50 con 0x7f1fe014be90 2024-03-25T18:16:17.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.358+0000 7f1fde59c640 10 monclient: get_auth_request con 0x7f1fc006ddf0 auth_method 0 2024-03-25T18:16:17.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.358+0000 7f1fceffd640 10 client.4721.objecter ms_handle_connect 0x7f1fc006ddf0 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f1fd80537d0 con 0x7f1fe014be90 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 10 client.4721.objecter ms_dispatch 0x7f1fe0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 3 client.4721.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 3 client.4721.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fd804f2e0 con 0x7f1fe014be90 2024-03-25T18:16:17.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.364+0000 7f1fceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:17.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.585+0000 7f1fe4e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:17.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.585+0000 7f1fe4e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:17.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.585+0000 7f1fe4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-25T18:16:17.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.585+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe01456f0 con 0x7f1fe014be90 2024-03-25T18:16:17.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.629+0000 7f1fceffd640 1 -- 172.21.15.84:0/2311358695 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f1fe01456f0 con 0x7f1fe014be90 2024-03-25T18:16:17.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.629+0000 7f1fceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:17.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.629+0000 7f1fceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:17.631 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:17.631 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.632+0000 7f1fe4e55640 10 librados: watch_flush enter 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 10 librados: watch_flush exit 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 10 client.4721.objecter successfully canceled tick 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 >> v1:172.21.15.157:6800/1801259026 conn(0x7f1fc006ddf0 legacy=0x7f1fc00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 10 monclient: shutdown 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 >> v1:172.21.15.84:6790/0 conn(0x7f1fe014be90 legacy=0x7f1fe012f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 shutdown_connections 2024-03-25T18:16:17.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 >> 172.21.15.84:0/2311358695 conn(0x7f1fe00617a0 msgr2=0x7f1fe00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 shutdown_connections 2024-03-25T18:16:17.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 -- 172.21.15.84:0/2311358695 wait complete. 2024-03-25T18:16:17.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:17.633+0000 7f1fe4e55640 1 librados: shutdown 2024-03-25T18:16:17.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:24.451 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:24.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.566+0000 7fb26c36d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.566+0000 7fb26c36d640 10 monclient: build_initial_monmap 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.566+0000 7fb26c36d640 10 monclient: monmap: 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:24.567962+0000 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:24.567962+0000 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:24.568 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:24.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:24.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.567+0000 7fb26c36d640 1 Processor -- start 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.567+0000 7fb26c36d640 1 -- start start 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.567+0000 7fb26c36d640 10 monclient: init 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2640a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-a con 0x7fb26413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2641430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:24.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb264142520 con 0x7fb26413f710 2024-03-25T18:16:24.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2640b28a0 con 0x7fb2641430c0 2024-03-25T18:16:24.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2640b2a30 con 0x7fb2640a98f0 2024-03-25T18:16:24.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:24.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.568+0000 7fb26c36d640 10 monclient(hunting): authenticate will time out at 7628.867292s 2024-03-25T18:16:24.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.583+0000 7fb26a109640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb2640a98f0 0x7fb2640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33494/0 (socket says 172.21.15.84:33494) 2024-03-25T18:16:24.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.583+0000 7fb26a109640 1 -- 172.21.15.84:0/2577107235 learned_addr learned my addr 172.21.15.84:0/2577107235 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:24.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500867838 0 0) 0x7fb2640b28a0 con 0x7fb2641430c0 2024-03-25T18:16:24.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:16:24.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.601+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250003520 con 0x7fb2641430c0 2024-03-25T18:16:24.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255736394 0 0) 0x7fb2640b2a30 con 0x7fb2640a98f0 2024-03-25T18:16:24.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:16:24.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.603+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2640b28a0 con 0x7fb2640a98f0 2024-03-25T18:16:24.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877834885 0 0) 0x7fb2640b28a0 con 0x7fb2640a98f0 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 >> v1:172.21.15.157:6789/0 conn(0x7fb2641430c0 legacy=0x7fb26414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 >> v1:172.21.15.84:6789/0 conn(0x7fb26413f710 legacy=0x7fb2641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 10 monclient: _finish_hunting 0 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 1 monclient: found mon.noname-b 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2640b44c0 con 0x7fb2640a98f0 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 10 monclient: _finish_auth 0 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 10 monclient: _check_auth_tickets 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb269107640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb26c36d640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:16:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.611+0000 7fb26c36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.614+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb260001630 con 0x7fb2640a98f0 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.614+0000 7fb269107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.614+0000 7fb269107640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.614+0000 7fb269107640 10 monclient: dump: 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:24.615 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.614+0000 7fb26c36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:24.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2600018c0 con 0x7fb2640a98f0 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 1 -- 172.21.15.84:0/2577107235 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb260001c50 con 0x7fb2640a98f0 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb269107640 10 monclient: dump: 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:24.619 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:24.620 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb26c36d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:24.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.617+0000 7fb26c36d640 4 set_mon_vals no callback set 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 10 monclient: shutdown 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 1 -- 172.21.15.84:0/2577107235 >> v1:172.21.15.84:6790/0 conn(0x7fb2640a98f0 legacy=0x7fb2640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 1 -- 172.21.15.84:0/2577107235 shutdown_connections 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 1 -- 172.21.15.84:0/2577107235 >> 172.21.15.84:0/2577107235 conn(0x7fb2640b7ea0 msgr2=0x7fb2640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 1 -- 172.21.15.84:0/2577107235 shutdown_connections 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.618+0000 7fb26c36d640 1 -- 172.21.15.84:0/2577107235 wait complete. 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 10 monclient: build_initial_monmap 2024-03-25T18:16:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 10 monclient: monmap: 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 librados: starting msgr at 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 librados: starting objecter 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 Processor -- start 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 -- start start 2024-03-25T18:16:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 librados: setting wanted keys 2024-03-25T18:16:24.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 1 librados: calling monclient init 2024-03-25T18:16:24.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.619+0000 7fb26c36d640 10 monclient: init 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2640a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-a con 0x7fb26413f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2641430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2641d41e0 con 0x7fb26413f710 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2641d44a0 con 0x7fb2641430c0 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb264139a00 con 0x7fb2640a98f0 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:24.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.620+0000 7fb26c36d640 10 monclient(hunting): authenticate will time out at 7628.919637s 2024-03-25T18:16:24.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.634+0000 7fb269908640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb26413f710 0x7fb2641d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:45442/0 (socket says 172.21.15.84:45442) 2024-03-25T18:16:24.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.634+0000 7fb269908640 1 -- 172.21.15.84:0/2838259716 learned_addr learned my addr 172.21.15.84:0/2838259716 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:24.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.638+0000 7fb25affd640 10 client.?.objecter ms_handle_connect 0x7fb2640a98f0 2024-03-25T18:16:24.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.638+0000 7fb25affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:24.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.640+0000 7fb25affd640 10 client.?.objecter ms_handle_connect 0x7fb26413f710 2024-03-25T18:16:24.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.640+0000 7fb25affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:24.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.641+0000 7fb25affd640 10 client.?.objecter ms_handle_connect 0x7fb2641430c0 2024-03-25T18:16:24.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.641+0000 7fb25affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398645981 0 0) 0x7fb2641d41e0 con 0x7fb26413f710 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:16:24.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.645+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb248004eb0 con 0x7fb26413f710 2024-03-25T18:16:24.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805864311 0 0) 0x7fb264139a00 con 0x7fb2640a98f0 2024-03-25T18:16:24.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.652+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2641d41e0 con 0x7fb2640a98f0 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852433726 0 0) 0x7fb2641d44a0 con 0x7fb2641430c0 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.653+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb264139a00 con 0x7fb2641430c0 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224698919 0 0) 0x7fb248004eb0 con 0x7fb26413f710 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2641d44a0 con 0x7fb26413f710 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb2540019d0 con 0x7fb26413f710 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:24.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 10 monclient(hunting): dump: 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.654+0000 7fb25affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3989271761 0 0) 0x7fb2641d44a0 con 0x7fb26413f710 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 >> v1:172.21.15.84:6790/0 conn(0x7fb2640a98f0 legacy=0x7fb2641d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 >> v1:172.21.15.157:6789/0 conn(0x7fb2641430c0 legacy=0x7fb2641384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 1 monclient: found mon.a 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2641d7810 con 0x7fb26413f710 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:24.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb25affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 10 monclient: _renew_subs 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2641d8440 con 0x7fb26413f710 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 10 monclient: _renew_subs 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2641d51e0 con 0x7fb26413f710 2024-03-25T18:16:24.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.662+0000 7fb26c36d640 1 librados: init done 2024-03-25T18:16:24.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.663+0000 7fb26c36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:24.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.663+0000 7fb26c36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:24.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.663+0000 7fb26c36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:24.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.663+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2640627b0 con 0x7fb26413f710 2024-03-25T18:16:24.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb254001b30 con 0x7fb26413f710 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fb254003a60 con 0x7fb26413f710 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 10 monclient: dump: 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb268906640 4 set_mon_vals no callback set 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.968+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fb254003c80 con 0x7fb26413f710 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.969+0000 7fb26a109640 10 monclient: get_auth_request con 0x7fb24806ddf0 auth_method 0 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.969+0000 7fb25affd640 10 client.4734.objecter ms_handle_connect 0x7fb24806ddf0 2024-03-25T18:16:24.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7fb254054850 con 0x7fb26413f710 2024-03-25T18:16:24.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 10 client.4734.objecter ms_dispatch 0x7fb264000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:24.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 3 client.4734.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:24.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 3 client.4734.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:24.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T18:16:24.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb254050360 con 0x7fb26413f710 2024-03-25T18:16:24.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:24.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:24.976+0000 7fb25affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:25.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.187+0000 7fb26c36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.187+0000 7fb26c36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.187+0000 7fb26c36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:25.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.187+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb264141c20 con 0x7fb26413f710 2024-03-25T18:16:25.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.285+0000 7fb25affd640 1 -- 172.21.15.84:0/2838259716 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7fb254052290 con 0x7fb26413f710 2024-03-25T18:16:25.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.285+0000 7fb25affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.285+0000 7fb25affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:25.287 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:25.287 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.287+0000 7fb26c36d640 10 librados: watch_flush enter 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 10 librados: watch_flush exit 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 10 client.4734.objecter successfully canceled tick 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 >> v1:172.21.15.157:6800/1801259026 conn(0x7fb24806ddf0 legacy=0x7fb2480702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 10 monclient: shutdown 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 >> v1:172.21.15.84:6789/0 conn(0x7fb26413f710 legacy=0x7fb2641d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 shutdown_connections 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 >> 172.21.15.84:0/2838259716 conn(0x7fb2640b7ea0 msgr2=0x7fb264074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 shutdown_connections 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 -- 172.21.15.84:0/2838259716 wait complete. 2024-03-25T18:16:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:25.288+0000 7fb26c36d640 1 librados: shutdown 2024-03-25T18:16:25.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:26.385 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:23.204 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.389 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:23.208 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.398 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:23.181 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.399 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:23.218 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.402 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:23.186 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.404 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:23.223 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.414 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:23.233 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.422 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:23.206 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:26.425 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:23.243 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.497 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:26.316 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.543 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:26.362 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.545 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:26.328 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.562 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:26.346 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.588 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:26.407 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.591 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:26.375 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.600 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:26.419 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.612 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:26.431 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.633 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:26.452 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:32.107 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:32.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.219+0000 7f2fe54a1640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.219+0000 7f2fe54a1640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.219+0000 7f2fe54a1640 10 monclient: monmap: 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:32.220578+0000 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:32.220578+0000 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:32.221 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.220+0000 7f2fe54a1640 1 Processor -- start 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.220+0000 7f2fe54a1640 1 -- start start 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.220+0000 7f2fe54a1640 10 monclient: init 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe00a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-b con 0x7f2fe013f710 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-c con 0x7f2fe01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe0142520 con 0x7f2fe00a98f0 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe00b28a0 con 0x7f2fe01430c0 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe00b2a30 con 0x7f2fe013f710 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.221+0000 7f2fe54a1640 10 monclient(hunting): authenticate will time out at 7636.520186s 2024-03-25T18:16:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.226+0000 7f2fdf7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2fe01430c0 0x7f2fe014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:36158/0 (socket says 172.21.15.84:36158) 2024-03-25T18:16:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.226+0000 7f2fdf7fe640 1 -- 172.21.15.84:0/1166257028 learned_addr learned my addr 172.21.15.84:0/1166257028 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678728950 0 0) 0x7f2fe00b28a0 con 0x7f2fe01430c0 2024-03-25T18:16:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:16:32.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.236+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc8003520 con 0x7f2fe01430c0 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100456137 0 0) 0x7f2fe00b2a30 con 0x7f2fe013f710 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:16:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.241+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe00b28a0 con 0x7f2fe013f710 2024-03-25T18:16:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428594027 0 0) 0x7f2fc8003520 con 0x7f2fe01430c0 2024-03-25T18:16:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 >> v1:172.21.15.84:6790/0 conn(0x7f2fe013f710 legacy=0x7f2fe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 >> v1:172.21.15.84:6789/0 conn(0x7f2fe00a98f0 legacy=0x7f2fe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 monclient: found mon.noname-c 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe00b44c0 con 0x7f2fe01430c0 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2fd8001a10 con 0x7f2fe01430c0 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.242+0000 7f2fddffb640 10 monclient: dump: 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.245 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.243+0000 7f2fe54a1640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:16:32.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.243+0000 7f2fe54a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.1 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fd8001bf0 con 0x7f2fe01430c0 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 1 -- 172.21.15.84:0/1166257028 <== mon.1 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2fd8002000 con 0x7f2fe01430c0 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fddffb640 10 monclient: dump: 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:32.253 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:32.254 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:32.254 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:32.254 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fe54a1640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.251+0000 7f2fe54a1640 4 set_mon_vals no callback set 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 10 monclient: shutdown 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1166257028 >> v1:172.21.15.157:6789/0 conn(0x7f2fe01430c0 legacy=0x7f2fe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1166257028 shutdown_connections 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1166257028 >> 172.21.15.84:0/1166257028 conn(0x7f2fe00b7ea0 msgr2=0x7f2fe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1166257028 shutdown_connections 2024-03-25T18:16:32.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.252+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1166257028 wait complete. 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.253+0000 7f2fe54a1640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.253+0000 7f2fe54a1640 10 monclient: monmap: 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 librados: starting msgr at 2024-03-25T18:16:32.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 librados: starting objecter 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 Processor -- start 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 -- start start 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 librados: setting wanted keys 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 1 librados: calling monclient init 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.254+0000 7f2fe54a1640 10 monclient: init 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-c con 0x7f2fe00a98f0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 10 monclient(hunting): picked mon.noname-b con 0x7f2fe01430c0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe00715c0 con 0x7f2fe013f710 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.255+0000 7f2fe54a1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe006fa20 con 0x7f2fe01430c0 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.256+0000 7f2fe54a1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe006fd00 con 0x7f2fe00a98f0 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.256+0000 7f2fe54a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.256+0000 7f2fe54a1640 10 monclient(hunting): authenticate will time out at 7636.555105s 2024-03-25T18:16:32.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.263+0000 7f2fdf7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2fe01430c0 0x7f2fe014c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:36216/0 (socket says 172.21.15.84:36216) 2024-03-25T18:16:32.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.263+0000 7f2fdf7fe640 1 -- 172.21.15.84:0/1296465930 learned_addr learned my addr 172.21.15.84:0/1296465930 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:32.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.266+0000 7f2fc77fe640 10 client.?.objecter ms_handle_connect 0x7f2fe01430c0 2024-03-25T18:16:32.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.266+0000 7f2fc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467477422 0 0) 0x7f2fe006fa20 con 0x7f2fe01430c0 2024-03-25T18:16:32.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:16:32.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.271+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc004eb0 con 0x7f2fe01430c0 2024-03-25T18:16:32.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.275+0000 7f2fc77fe640 10 client.?.objecter ms_handle_connect 0x7f2fe00a98f0 2024-03-25T18:16:32.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.275+0000 7f2fc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.279+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321039558 0 0) 0x7f2fcc004eb0 con 0x7f2fe01430c0 2024-03-25T18:16:32.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.279+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe006fa20 con 0x7f2fe01430c0 2024-03-25T18:16:32.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.284+0000 7f2fc77fe640 10 client.?.objecter ms_handle_connect 0x7f2fe013f710 2024-03-25T18:16:32.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.284+0000 7f2fc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760905697 0 0) 0x7f2fe006fd00 con 0x7f2fe00a98f0 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:16:32.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.286+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc004eb0 con 0x7f2fe00a98f0 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.290+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2fd80020c0 con 0x7f2fe01430c0 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.290+0000 7f2fc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.290+0000 7f2fc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.290+0000 7f2fc77fe640 10 monclient(hunting): dump: 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:32.292 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:32.293 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.290+0000 7f2fc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 388144204 0 0) 0x7f2fe006fa20 con 0x7f2fe01430c0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 >> v1:172.21.15.84:6790/0 conn(0x7f2fe00a98f0 legacy=0x7f2fe006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 >> v1:172.21.15.84:6789/0 conn(0x7f2fe013f710 legacy=0x7f2fe014bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 1 monclient: found mon.b 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe006ffe0 con 0x7f2fe01430c0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 10 monclient: _renew_subs 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe00702c0 con 0x7f2fe01430c0 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 10 monclient: _renew_subs 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe014cd20 con 0x7f2fe01430c0 2024-03-25T18:16:32.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.297+0000 7f2fe54a1640 1 librados: init done 2024-03-25T18:16:32.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.298+0000 7f2fe54a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.298+0000 7f2fe54a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.298+0000 7f2fe54a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.298+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe00627b0 con 0x7f2fe01430c0 2024-03-25T18:16:32.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fd8004e10 con 0x7f2fe01430c0 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f2fd8005120 con 0x7f2fe01430c0 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fc77fe640 10 monclient: dump: 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:32.584 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.582+0000 7f2fdd7fa640 4 set_mon_vals no callback set 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.583+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f2fd801b170 con 0x7f2fe01430c0 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.584+0000 7f2fdeffd640 10 monclient: get_auth_request con 0x7f2fcc06dc50 auth_method 0 2024-03-25T18:16:32.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.584+0000 7f2fc77fe640 10 client.4741.objecter ms_handle_connect 0x7f2fcc06dc50 2024-03-25T18:16:32.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.591+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f2fd8054e30 con 0x7f2fe01430c0 2024-03-25T18:16:32.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.591+0000 7f2fc77fe640 10 client.4741.objecter ms_dispatch 0x7f2fe0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:32.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.591+0000 7f2fc77fe640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:32.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.591+0000 7f2fc77fe640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:32.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.591+0000 7f2fc77fe640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T18:16:32.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.592+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fd8050770 con 0x7f2fe01430c0 2024-03-25T18:16:32.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.592+0000 7f2fc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.592+0000 7f2fc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:32.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.833+0000 7f2fe54a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.833+0000 7f2fe54a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.833+0000 7f2fe54a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.833+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fe013df00 con 0x7f2fe01430c0 2024-03-25T18:16:32.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.929+0000 7f2fc77fe640 1 -- 172.21.15.84:0/1296465930 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f2fd8052870 con 0x7f2fe01430c0 2024-03-25T18:16:32.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.929+0000 7f2fc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:32.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.929+0000 7f2fc77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:32.931 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:32.931 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:32.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 10 librados: watch_flush enter 2024-03-25T18:16:32.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 10 librados: watch_flush exit 2024-03-25T18:16:32.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T18:16:32.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 10 client.4741.objecter successfully canceled tick 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 >> v1:172.21.15.157:6800/1801259026 conn(0x7f2fcc06dc50 legacy=0x7f2fcc070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 10 monclient: shutdown 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 >> v1:172.21.15.157:6789/0 conn(0x7f2fe01430c0 legacy=0x7f2fe014c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 shutdown_connections 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.931+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 >> 172.21.15.84:0/1296465930 conn(0x7f2fe00b7ea0 msgr2=0x7f2fe0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.932+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 shutdown_connections 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.932+0000 7f2fe54a1640 1 -- 172.21.15.84:0/1296465930 wait complete. 2024-03-25T18:16:32.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:32.932+0000 7f2fe54a1640 1 librados: shutdown 2024-03-25T18:16:32.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:39.751 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.860+0000 7f403800f640 10 monclient: get_monmap_and_config 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.860+0000 7f403800f640 10 monclient: build_initial_monmap 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.860+0000 7f403800f640 10 monclient: monmap: 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:39.861429+0000 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:39.861429+0000 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:39.861 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.860+0000 7f403800f640 1 Processor -- start 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 1 -- start start 2024-03-25T18:16:39.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient: init 2024-03-25T18:16:39.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:39.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient(hunting): picked mon.noname-c con 0x7f40300a98f0 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient(hunting): picked mon.noname-a con 0x7f403013f710 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient(hunting): picked mon.noname-b con 0x7f40301430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030142520 con 0x7f403013f710 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b28a0 con 0x7f40300a98f0 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b2a30 con 0x7f40301430c0 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.861+0000 7f403800f640 10 monclient(hunting): authenticate will time out at 7644.160795s 2024-03-25T18:16:39.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.863+0000 7f40355aa640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f403013f710 0x7f40301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44918/0 (socket says 172.21.15.84:44918) 2024-03-25T18:16:39.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.863+0000 7f40355aa640 1 -- 172.21.15.84:0/878141405 learned_addr learned my addr 172.21.15.84:0/878141405 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:39.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890614314 0 0) 0x7f4030142520 con 0x7f403013f710 2024-03-25T18:16:39.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:39.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:39.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:39.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:16:39.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.868+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c003520 con 0x7f403013f710 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3426629008 0 0) 0x7f401c003520 con 0x7f403013f710 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 >> v1:172.21.15.84:6790/0 conn(0x7f40301430c0 legacy=0x7f403014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 >> v1:172.21.15.157:6789/0 conn(0x7f40300a98f0 legacy=0x7f40300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 1 monclient: found mon.noname-a 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300b44c0 con 0x7f403013f710 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 10 monclient: _finish_auth 0 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f4034da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:39.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f403800f640 5 monclient: authenticate success, global_id 4743 2024-03-25T18:16:39.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.874+0000 7f403800f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.876+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4020001690 con 0x7f403013f710 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.876+0000 7f4034da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.876+0000 7f4034da9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.876+0000 7f4034da9640 10 monclient: dump: 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:39.878 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:39.879 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:39.879 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:39.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.876+0000 7f403800f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:39.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4020001960 con 0x7f403013f710 2024-03-25T18:16:39.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:39.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 1 -- 172.21.15.84:0/878141405 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f4020001cf0 con 0x7f403013f710 2024-03-25T18:16:39.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f4034da9640 10 monclient: dump: 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:39.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 4 set_mon_vals no callback set 2024-03-25T18:16:39.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 10 monclient: shutdown 2024-03-25T18:16:39.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 1 -- 172.21.15.84:0/878141405 >> v1:172.21.15.84:6789/0 conn(0x7f403013f710 legacy=0x7f40301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.885+0000 7f403800f640 1 -- 172.21.15.84:0/878141405 shutdown_connections 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.886+0000 7f403800f640 1 -- 172.21.15.84:0/878141405 >> 172.21.15.84:0/878141405 conn(0x7f40300b7ea0 msgr2=0x7f40300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.886+0000 7f403800f640 1 -- 172.21.15.84:0/878141405 shutdown_connections 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.886+0000 7f403800f640 1 -- 172.21.15.84:0/878141405 wait complete. 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.886+0000 7f403800f640 10 monclient: build_initial_monmap 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.886+0000 7f403800f640 10 monclient: monmap: 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:39.890 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 librados: starting msgr at 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 librados: starting objecter 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 Processor -- start 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 -- start start 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 librados: setting wanted keys 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 librados: calling monclient init 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient: init 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient(hunting): picked mon.noname-a con 0x7f40300a98f0 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient(hunting): picked mon.noname-b con 0x7f403013f710 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient(hunting): picked mon.noname-c con 0x7f40301430c0 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b6440 con 0x7f40300a98f0 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b6720 con 0x7f403013f710 2024-03-25T18:16:39.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b78a0 con 0x7f40301430c0 2024-03-25T18:16:39.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:39.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.887+0000 7f403800f640 10 monclient(hunting): authenticate will time out at 7644.186836s 2024-03-25T18:16:39.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.895+0000 7f4035dab640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f40300a98f0 0x7f40300b7190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44964/0 (socket says 172.21.15.84:44964) 2024-03-25T18:16:39.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.895+0000 7f4035dab640 1 -- 172.21.15.84:0/1507041624 learned_addr learned my addr 172.21.15.84:0/1507041624 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:39.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.896+0000 7f40267fc640 10 client.?.objecter ms_handle_connect 0x7f40300a98f0 2024-03-25T18:16:39.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.896+0000 7f40267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:39.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.902+0000 7f40267fc640 10 client.?.objecter ms_handle_connect 0x7f40301430c0 2024-03-25T18:16:39.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.902+0000 7f40267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:39.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.904+0000 7f40267fc640 10 client.?.objecter ms_handle_connect 0x7f403013f710 2024-03-25T18:16:39.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.904+0000 7f40267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036912372 0 0) 0x7f40300b6440 con 0x7f40300a98f0 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:16:39.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.907+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4014004eb0 con 0x7f40300a98f0 2024-03-25T18:16:39.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567122855 0 0) 0x7f40300b78a0 con 0x7f40301430c0 2024-03-25T18:16:39.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:39.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:39.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:39.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:16:39.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.911+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40300b6440 con 0x7f40301430c0 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2386376541 0 0) 0x7f4014004eb0 con 0x7f40300a98f0 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40300b78a0 con 0x7f40300a98f0 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f402c001960 con 0x7f40300a98f0 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 10 monclient(hunting): dump: 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:39.921 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:39.922 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:39.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.919+0000 7f40267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:39.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767678235 0 0) 0x7f40300b6720 con 0x7f403013f710 2024-03-25T18:16:39.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:39.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:39.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:39.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:16:39.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.923+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4014004eb0 con 0x7f403013f710 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 308745075 0 0) 0x7f40300b78a0 con 0x7f40300a98f0 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 >> v1:172.21.15.84:6790/0 conn(0x7f40301430c0 legacy=0x7f40301d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 >> v1:172.21.15.157:6789/0 conn(0x7f403013f710 legacy=0x7f40301d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 1 monclient: found mon.a 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403013bb30 con 0x7f40300a98f0 2024-03-25T18:16:39.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f40267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 5 monclient: authenticate success, global_id 4746 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 10 monclient: _renew_subs 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40301e2fe0 con 0x7f40300a98f0 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 10 client.4746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 10 monclient: _renew_subs 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40301e3350 con 0x7f40300a98f0 2024-03-25T18:16:39.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.928+0000 7f403800f640 1 librados: init done 2024-03-25T18:16:39.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.929+0000 7f403800f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:39.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.929+0000 7f403800f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:39.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.929+0000 7f403800f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:39.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:39.929+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40301d7590 con 0x7f40300a98f0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f402c0035f0 con 0x7f40300a98f0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7f402c003940 con 0x7f40300a98f0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f40267fc640 10 monclient: dump: 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:40.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.238+0000 7f4027fff640 4 set_mon_vals no callback set 2024-03-25T18:16:40.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.239+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7f402c003b60 con 0x7f40300a98f0 2024-03-25T18:16:40.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.240+0000 7f40355aa640 10 monclient: get_auth_request con 0x7f401406de00 auth_method 0 2024-03-25T18:16:40.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.240+0000 7f40267fc640 10 client.4746.objecter ms_handle_connect 0x7f401406de00 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7f402c0547a0 con 0x7f40300a98f0 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 10 client.4746.objecter ms_dispatch 0x7f4030000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 3 client.4746.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 3 client.4746.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 20 client.4746.objecter dump_active .. 0 homeless 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.245+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f402c0502b0 con 0x7f40300a98f0 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.246+0000 7f40267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.246+0000 7f40267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:40.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.474+0000 7f403800f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.474+0000 7f403800f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.474+0000 7f403800f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-25T18:16:40.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.474+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40300627b0 con 0x7f40300a98f0 2024-03-25T18:16:40.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.522+0000 7f40267fc640 1 -- 172.21.15.84:0/1507041624 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7f402c0521e0 con 0x7f40300a98f0 2024-03-25T18:16:40.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.522+0000 7f40267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.522+0000 7f40267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:40.523 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:40.524 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:40.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 10 librados: watch_flush enter 2024-03-25T18:16:40.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 10 librados: watch_flush exit 2024-03-25T18:16:40.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 20 client.4746.objecter shutdown clearing up homeless session... 2024-03-25T18:16:40.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 10 client.4746.objecter successfully canceled tick 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 >> v1:172.21.15.157:6800/1801259026 conn(0x7f401406de00 legacy=0x7f40140702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 10 monclient: shutdown 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 >> v1:172.21.15.84:6789/0 conn(0x7f40300a98f0 legacy=0x7f40300b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 shutdown_connections 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 >> 172.21.15.84:0/1507041624 conn(0x7f40300b7ea0 msgr2=0x7f4030074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 shutdown_connections 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 -- 172.21.15.84:0/1507041624 wait complete. 2024-03-25T18:16:40.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:40.524+0000 7f403800f640 1 librados: shutdown 2024-03-25T18:16:40.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:47.345 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.462+0000 7fc82936d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.462+0000 7fc82936d640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.462+0000 7fc82936d640 10 monclient: monmap: 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:16:47.463145+0000 2024-03-25T18:16:47.463 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:16:47.463145+0000 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-25T18:16:47.464 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.463+0000 7fc82936d640 1 Processor -- start 2024-03-25T18:16:47.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.463+0000 7fc82936d640 1 -- start start 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.463+0000 7fc82936d640 10 monclient: init 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8240b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8240b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-a con 0x7fc82414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240af940 con 0x7fc82414be90 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240b0a70 con 0x7fc8240b1730 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240b0d50 con 0x7fc8240b4d60 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.464+0000 7fc82936d640 10 monclient(hunting): authenticate will time out at 7651.762346s 2024-03-25T18:16:47.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.475+0000 7fc822ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc8240b4d60 0x7fc82413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36960/0 (socket says 172.21.15.84:36960) 2024-03-25T18:16:47.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.475+0000 7fc822ffd640 1 -- 172.21.15.84:0/905366561 learned_addr learned my addr 172.21.15.84:0/905366561 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:47.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190368071 0 0) 0x7fc8240b0d50 con 0x7fc8240b4d60 2024-03-25T18:16:47.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:16:47.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.487+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc80c003520 con 0x7fc8240b4d60 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310104257 0 0) 0x7fc8240b0a70 con 0x7fc8240b1730 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:16:47.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.498+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8240b0d50 con 0x7fc8240b1730 2024-03-25T18:16:47.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640525939 0 0) 0x7fc8240af940 con 0x7fc82414be90 2024-03-25T18:16:47.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:16:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.503+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8240b0a70 con 0x7fc82414be90 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557861903 0 0) 0x7fc80c003520 con 0x7fc8240b4d60 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 >> v1:172.21.15.157:6789/0 conn(0x7fc8240b1730 legacy=0x7fc8240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 >> v1:172.21.15.84:6789/0 conn(0x7fc82414be90 legacy=0x7fc82414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 monclient: found mon.noname-b 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8240b1030 con 0x7fc8240b4d60 2024-03-25T18:16:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc814001690 con 0x7fc8240b4d60 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc821ffb640 10 monclient: dump: 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:47.506 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:47.507 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc82936d640 5 monclient: authenticate success, global_id 4739 2024-03-25T18:16:47.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.504+0000 7fc82936d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:47.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8140017f0 con 0x7fc8240b4d60 2024-03-25T18:16:47.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:47.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 1 -- 172.21.15.84:0/905366561 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc814001c00 con 0x7fc8240b4d60 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc821ffb640 10 monclient: dump: 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc82936d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.521+0000 7fc82936d640 4 set_mon_vals no callback set 2024-03-25T18:16:47.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 10 monclient: shutdown 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 1 -- 172.21.15.84:0/905366561 >> v1:172.21.15.84:6790/0 conn(0x7fc8240b4d60 legacy=0x7fc82413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 1 -- 172.21.15.84:0/905366561 shutdown_connections 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 1 -- 172.21.15.84:0/905366561 >> 172.21.15.84:0/905366561 conn(0x7fc8240617a0 msgr2=0x7fc824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 1 -- 172.21.15.84:0/905366561 shutdown_connections 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 1 -- 172.21.15.84:0/905366561 wait complete. 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.522+0000 7fc82936d640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 10 monclient: monmap: 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.525 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.noname-b 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 librados: starting msgr at 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 librados: starting objecter 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 Processor -- start 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 -- start start 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 librados: setting wanted keys 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 1 librados: calling monclient init 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.523+0000 7fc82936d640 10 monclient: init 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8240b1730 addr v1:172.21.15.157:6789/0 2024-03-25T18:16:47.526 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8240b4d60 addr v1:172.21.15.84:6790/0 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient(hunting): picked mon.noname-a con 0x7fc82414be90 addr v1:172.21.15.84:6789/0 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8241d7980 con 0x7fc82414be90 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8241420a0 con 0x7fc8240b1730 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8241432c0 con 0x7fc8240b4d60 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.524+0000 7fc82936d640 10 monclient(hunting): authenticate will time out at 7651.822399s 2024-03-25T18:16:47.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.535+0000 7fc822ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc8240b1730 0x7fc824141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.84:33404/0 (socket says 172.21.15.84:33404) 2024-03-25T18:16:47.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.535+0000 7fc822ffd640 1 -- 172.21.15.84:0/194529518 learned_addr learned my addr 172.21.15.84:0/194529518 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-25T18:16:47.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.536+0000 7fc80b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8240b1730 2024-03-25T18:16:47.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.536+0000 7fc80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.541+0000 7fc80b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8240b4d60 2024-03-25T18:16:47.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.541+0000 7fc80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.548+0000 7fc80b7fe640 10 client.?.objecter ms_handle_connect 0x7fc82414be90 2024-03-25T18:16:47.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.548+0000 7fc80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058437824 0 0) 0x7fc8241420a0 con 0x7fc8240b1730 2024-03-25T18:16:47.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:16:47.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.549+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc810004eb0 con 0x7fc8240b1730 2024-03-25T18:16:47.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486460579 0 0) 0x7fc8241432c0 con 0x7fc8240b4d60 2024-03-25T18:16:47.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:16:47.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.553+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8241420a0 con 0x7fc8240b4d60 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 792757913 0 0) 0x7fc810004eb0 con 0x7fc8240b1730 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8241432c0 con 0x7fc8240b1730 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc814001bc0 con 0x7fc8240b1730 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.561 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548035388 0 0) 0x7fc8241d7980 con 0x7fc82414be90 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 10 monclient(hunting): my global_id is 4752 2024-03-25T18:16:47.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.560+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc810004eb0 con 0x7fc82414be90 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2457606381 0 0) 0x7fc8241420a0 con 0x7fc8240b4d60 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8241d7980 con 0x7fc8240b4d60 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc818001970 con 0x7fc8240b4d60 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.573+0000 7fc80b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:47.574 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:47.575 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:47.575 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661279865 0 0) 0x7fc8241432c0 con 0x7fc8240b1730 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 >> v1:172.21.15.84:6790/0 conn(0x7fc8240b4d60 legacy=0x7fc824138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 >> v1:172.21.15.84:6789/0 conn(0x7fc82414be90 legacy=0x7fc8241d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 1 monclient: found mon.b 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc824143940 con 0x7fc8240b1730 2024-03-25T18:16:47.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc80b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 5 monclient: authenticate success, global_id 4750 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 10 monclient: _renew_subs 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc824138a70 con 0x7fc8240b1730 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 10 client.4750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 10 monclient: _renew_subs 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8241d8700 con 0x7fc8240b1730 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.574+0000 7fc82936d640 1 librados: init done 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.575+0000 7fc82936d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.575+0000 7fc82936d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.575+0000 7fc82936d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:47.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.575+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824143130 con 0x7fc8240b1730 2024-03-25T18:16:47.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8140011f0 con 0x7fc8240b1730 2024-03-25T18:16:47.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:47.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 30392175 0 0) 0x7fc814004cb0 con 0x7fc8240b1730 2024-03-25T18:16:47.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 10 monclient: dump: 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 790a4795-8265-4274-9bee-7eca8ec16c87 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-25T18:04:32.031466+0000 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 3 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.157:6789/0 mon.b 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-25T18:16:47.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc8217fa640 4 set_mon_vals no callback set 2024-03-25T18:16:47.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.887+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3135922277 0 0) 0x7fc81401acc0 con 0x7fc8240b1730 2024-03-25T18:16:47.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.888+0000 7fc8227fc640 10 monclient: get_auth_request con 0x7fc81006ded0 auth_method 0 2024-03-25T18:16:47.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.888+0000 7fc80b7fe640 10 client.4750.objecter ms_handle_connect 0x7fc81006ded0 2024-03-25T18:16:47.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1419153696 0 0) 0x7fc8140549e0 con 0x7fc8240b1730 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 10 client.4750.objecter ms_dispatch 0x7fc824000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 3 client.4750.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 3 client.4750.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 20 client.4750.objecter dump_active .. 0 homeless 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc814050320 con 0x7fc8240b1730 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:47.893+0000 7fc80b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:48.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.106+0000 7fc82936d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.106+0000 7fc82936d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.106+0000 7fc82936d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.157:6789/0 2024-03-25T18:16:48.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.106+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8241401f0 con 0x7fc8240b1730 2024-03-25T18:16:48.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.235+0000 7fc80b7fe640 1 -- 172.21.15.84:0/194529518 <== mon.1 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3106121235) 0x7fc814052420 con 0x7fc8240b1730 2024-03-25T18:16:48.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.235+0000 7fc80b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.235+0000 7fc80b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:48.236 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-25T18:16:48.236 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"790a4795-8265-4274-9bee-7eca8ec16c87","created":"2024-03-25T18:05:19.338970+0000","modified":"2024-03-25T18:15:24.077390+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:15:24.077390+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"784c9617-9da6-43f0-943e-f01177b02bba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9d216077-d072-4e6a-a53f-5f0c81125515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac1039e9-08a7-47d6-9e2a-571cfab09efd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c1ca3152-d5fb-4419-918f-0ed67b43351c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1ffdbcf6-1d36-40db-b94c-5e2ad4c9428b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dbf339f4-d0d7-4c48-b9d9-5dc391839d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f268061-9caa-483c-b5d3-b7f7b83b9b45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c150a5c7-47a4-49d5-aa05-058475f3559c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.237+0000 7fc82936d640 10 librados: watch_flush enter 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 10 librados: watch_flush exit 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 20 client.4750.objecter shutdown clearing up homeless session... 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 10 client.4750.objecter successfully canceled tick 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 >> v1:172.21.15.157:6800/1801259026 conn(0x7fc81006ded0 legacy=0x7fc810070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 10 monclient: shutdown 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 >> v1:172.21.15.157:6789/0 conn(0x7fc8240b1730 legacy=0x7fc824141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 shutdown_connections 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 >> 172.21.15.84:0/194529518 conn(0x7fc8240617a0 msgr2=0x7fc824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 shutdown_connections 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 -- 172.21.15.84:0/194529518 wait complete. 2024-03-25T18:16:48.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-25T18:16:48.238+0000 7fc82936d640 1 librados: shutdown 2024-03-25T18:16:48.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:48.250 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:16:48.252 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:16:48.252 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:16:48.252 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:49.612 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:46.431 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.619 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:46.437 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.665 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:11:46.484 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.666 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:46.485 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.668 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:46.451 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.680 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:46.499 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.687 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:46.470 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.707 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:11:46.527 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:49.714 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:11:46.497 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:50.322 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:11:25.002 35723== 2024-03-25T18:16:50.323 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:11:25.002 35723== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:50.361 INFO:tasks.ceph.osd.0.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:16:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:16:54.354 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:16:54.355 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:16:54.356 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:16:54.356 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:56.404 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:11:31.091 35721== 2024-03-25T18:16:56.404 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:11:31.091 35721== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:56.442 INFO:tasks.ceph.osd.1.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:00.458 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:17:00.458 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:17:00.459 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:17:00.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:02.267 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:11:36.942 35724== 2024-03-25T18:17:02.267 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:11:36.942 35724== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:02.314 INFO:tasks.ceph.osd.2.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:06.561 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:17:06.562 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:17:06.562 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:17:06.562 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:08.419 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:11:43.102 35722== 2024-03-25T18:17:08.419 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:11:43.102 35722== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:08.459 INFO:tasks.ceph.osd.3.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:09.628 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:06.447 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.632 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:06.451 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.664 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:06.483 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.680 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:06.463 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.683 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:06.503 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.688 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:06.507 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.688 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:06.471 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.711 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:06.495 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:09.712 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:06.531 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:12.664 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:17:12.665 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:17:12.665 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:17:12.665 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:14.784 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:11:49.470 35445== 2024-03-25T18:17:14.784 INFO:tasks.ceph.osd.4.smithi157.stderr:==00:00:11:49.470 35445== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:14.826 INFO:tasks.ceph.osd.4.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:18.767 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:17:18.768 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:17:18.768 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:17:18.768 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:20.579 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:11:55.245 35447== 2024-03-25T18:17:20.580 INFO:tasks.ceph.osd.5.smithi157.stderr:==00:00:11:55.245 35447== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:20.618 INFO:tasks.ceph.osd.5.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:24.870 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:17:24.870 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:17:24.871 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:17:24.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:26.607 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:12:01.273 35448== 2024-03-25T18:17:26.607 INFO:tasks.ceph.osd.6.smithi157.stderr:==00:00:12:01.273 35448== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:26.651 INFO:tasks.ceph.osd.6.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:29.631 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:26.450 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.678 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:26.497 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.692 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:26.511 35410== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.719 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:26.503 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.723 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:26.507 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.726 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:26.545 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.729 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:26.548 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.735 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:26.518 34901== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.740 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:26.559 35406== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:30.973 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:17:30.974 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:17:30.974 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:17:30.974 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:32.559 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:12:07.235 35446== 2024-03-25T18:17:32.559 INFO:tasks.ceph.osd.7.smithi157.stderr:==00:00:12:07.235 35446== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:32.588 INFO:tasks.ceph.osd.7.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:37.076 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:17:37.076 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:17:37.076 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:17:37.077 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:17:37.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:37.104 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:17:37.104 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:17:37.104 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:37.105 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:17:37.104+0000 7fdc2a664640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34899) UID: 0 2024-03-25T18:17:37.105 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-25T18:17:37.104+0000 7fdc2a664640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:17:37.133 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:17:37.133 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:17:37.133 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:17:37.133 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:37.160 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-03-25T18:17:37.147+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35375) UID: 0 2024-03-25T18:17:37.160 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-03-25T18:17:37.149+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:17:38.976 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:35.795 35406== 2024-03-25T18:17:38.976 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:12:35.795 35406== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:38.985 INFO:tasks.ceph.mon.a.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:43.235 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:17:43.235 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:17:43.236 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:17:43.236 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:43.252 INFO:tasks.ceph.mon.c.smithi084.stderr:2024-03-25T18:17:43.240+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35377) UID: 0 2024-03-25T18:17:43.253 INFO:tasks.ceph.mon.c.smithi084.stderr:2024-03-25T18:17:43.241+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:17:43.720 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:40.539 35410== 2024-03-25T18:17:43.720 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:12:40.539 35410== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:43.729 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:49.338 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:17:49.338 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:17:49.339 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:17:49.339 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:17:49.354 INFO:tasks.ceph.mon.b.smithi157.stderr:2024-03-25T18:17:49.342+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34897) UID: 0 2024-03-25T18:17:49.355 INFO:tasks.ceph.mon.b.smithi157.stderr:2024-03-25T18:17:49.344+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:17:49.790 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:46.574 34901== 2024-03-25T18:17:49.791 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:46.574 34901== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:17:49.800 INFO:tasks.ceph.mon.b.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:17:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:17:55.440 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:17:55.440 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:17:55.441 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:17:55.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:17:55.493 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:17:55.493 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:17:55.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:17:55.634 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:17:55.707 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:17:55.707 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:17:55.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:17:55.757 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:17:55.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:17:55.826 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:17:55.968 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:17:55.968 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:17:56.038 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:17:56.039 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:17:56.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:17:56.087 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:17:56.139 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:17:56.141 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/data/mon.a.tgz 2024-03-25T18:17:56.141 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-25T18:17:56.157 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.FTDLWvquUF 2024-03-25T18:17:56.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FTDLWvquUF 2024-03-25T18:17:56.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.FTDLWvquUF 2024-03-25T18:17:56.350 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.FTDLWvquUF is 132KB 2024-03-25T18:17:56.405 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.FTDLWvquUF 2024-03-25T18:17:56.420 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/data/mon.c.tgz 2024-03-25T18:17:56.420 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-25T18:17:56.474 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.MUCA9xx6Ke 2024-03-25T18:17:56.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MUCA9xx6Ke 2024-03-25T18:17:56.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.MUCA9xx6Ke 2024-03-25T18:17:56.646 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.MUCA9xx6Ke is 132KB 2024-03-25T18:17:56.698 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.MUCA9xx6Ke 2024-03-25T18:17:56.713 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/data/mon.b.tgz 2024-03-25T18:17:56.713 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-25T18:17:56.730 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.H4PGI5BRvg 2024-03-25T18:17:56.731 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.H4PGI5BRvg 2024-03-25T18:17:56.835 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.H4PGI5BRvg 2024-03-25T18:17:56.923 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.H4PGI5BRvg is 132KB 2024-03-25T18:17:56.975 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.H4PGI5BRvg 2024-03-25T18:17:56.989 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:17:56.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:17:56.992 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:17:57.380 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:17:57.656 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:17:57.658 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi084/crash 2024-03-25T18:17:57.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:17:57.687 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi157/crash 2024-03-25T18:17:57.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:17:57.715 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:17:57.715 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:17:57.733 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:17:57.806 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/osd.5.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35135.log 2024-03-25T18:17:57.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- 69.5% /var/log/ceph/ceph-client.admin.35255.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35095.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35135.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35346.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35255.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35095.log: gzip -5 87.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-osd.4.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.audit.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35095.log.gz 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-03-25T18:17:57.808 INFO:teuthology.orchestra.run.smithi157.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35297.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35175.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35297.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35175.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35297.log.gz 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.35215.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mgr.x.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35175.log.gz 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35215.log: -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-03-25T18:17:57.809 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35747.log 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35747.log: /var/log/ceph/ceph-client.admin.41648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41765.log 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42428.log 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2024-03-25T18:17:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42389.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41765.log: /var/log/ceph/ceph-client.admin.42428.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29565.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42389.log: gzip 86.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41765.log.gz -- 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.39345.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42428.log.gz 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.29565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29565.log.gz 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42389.log.gz 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41414.log 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-25T18:17:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41804.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41375.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41375.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41414.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41375.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40086.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:17:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.1.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41336.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42116.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41921.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41336.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41921.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42116.log.gz 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39969.log 2024-03-25T18:17:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40398.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41921.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40008.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41882.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39969.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40008.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.40398.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41453.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41609.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40008.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41999.log 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41882.log.gz 2024-03-25T18:17:57.819 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41609.log: gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.41453.log.gz --verbose 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40320.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41609.log.gz 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39501.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41999.log.gz 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35417.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40320.log.gz 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41531.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39501.log.gz 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35417.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:17:57.820 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35417.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41531.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40437.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42350.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39540.log: /var/log/ceph/ceph-client.admin.40437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42350.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40437.log.gz 2024-03-25T18:17:57.821 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39735.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42350.log.gz 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42506.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42506.log: /var/log/ceph/ceph-client.admin.39813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36776.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40281.log 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42506.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39657.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41726.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36776.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39657.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41726.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39657.log.gzgzip 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40359.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41726.log.gz 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-03-25T18:17:57.823 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39384.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40359.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42311.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40985.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42272.log 86.9% 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41258.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39263.log 2024-03-25T18:17:57.824 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42272.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39263.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41258.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.42155.log: -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39263.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40517.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz --verbose 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40125.log 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-25T18:17:57.825 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39044.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41960.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40125.log.gz/var/log/ceph/ceph-client.admin.39423.log: 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39852.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39044.log.gz 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40047.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41063.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41960.log.gz 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39852.log.gz 2024-03-25T18:17:57.826 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40047.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40673.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41063.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39774.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41492.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37775.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40242.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41492.log.gz 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-03-25T18:17:57.827 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37775.log: /var/log/ceph/ceph-client.admin.41102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40203.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42545.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37775.log.gz 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40203.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42038.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40203.log.gz 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42545.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42038.log: /var/log/ceph/ceph-client.admin.40634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41687.log 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42038.log.gz 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42545.log.gz 2024-03-25T18:17:57.828 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41687.log: /var/log/ceph/ceph-client.admin.39579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39696.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41687.log.gz 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42194.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39696.log: /var/log/ceph/ceph-osd.2.log: 92.9% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39696.log.gz 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-03-25T18:17:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.42194.log: /var/log/ceph/ceph-client.admin.40595.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42194.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35901.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41219.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35901.log: /var/log/ceph/ceph-client.admin.42467.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35901.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41024.log: /var/log/ceph/ceph-client.admin.39891.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-03-25T18:17:57.830 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39891.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39891.log.gz 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41570.log 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41843.log 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41570.log: gzip -5 86.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.39304.log 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41570.log.gz/var/log/ceph/ceph-client.admin.41843.log: 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39304.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41843.log.gz 2024-03-25T18:17:57.831 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-03-25T18:17:57.846 INFO:teuthology.orchestra.run.smithi157.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:17:58.595 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:17:58.628 INFO:teuthology.orchestra.run.smithi084.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:17:58.699 INFO:teuthology.orchestra.run.smithi157.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:17:58.722 INFO:teuthology.orchestra.run.smithi157.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:17:58.723 INFO:teuthology.orchestra.run.smithi157.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:17:58.882 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:17:58.916 INFO:teuthology.orchestra.run.smithi084.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:17:58.989 INFO:teuthology.orchestra.run.smithi157.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:18:02.862 INFO:teuthology.orchestra.run.smithi084.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:18:03.093 INFO:teuthology.orchestra.run.smithi157.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:18:03.095 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-25T18:18:03.095 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m5.318s 2024-03-25T18:18:03.095 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m9.005s 2024-03-25T18:18:03.095 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.340s 2024-03-25T18:18:03.215 INFO:teuthology.orchestra.run.smithi084.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:18:03.217 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-25T18:18:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m5.472s 2024-03-25T18:18:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m13.992s 2024-03-25T18:18:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.552s 2024-03-25T18:18:03.217 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:18:03.218 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi084/log 2024-03-25T18:18:03.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:18:05.641 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi157/log 2024-03-25T18:18:05.642 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:18:07.084 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:18:07.272 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6455f336aa5e4253acd336102057eee5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:18:07.275 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:18:07.285 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:18:07.309 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:18:07.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:18:07.323 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:18:07.405 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:18:07.415 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:18:07.415 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:18:07.418 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:18:07.432 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:18:07.435 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +552us[ +559us] +/- 92ms 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -1301us[-1294us] +/- 74ms 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +764us[ +770us] +/- 63ms 2024-03-25T18:18:07.437 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:18:07.440 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:18:07.440 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-25T18:18:07.440 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +558us[ +579us] +/- 92ms 2024-03-25T18:18:07.440 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -1298us[-1277us] +/- 74ms 2024-03-25T18:18:07.440 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +760us[ +781us] +/- 63ms 2024-03-25T18:18:07.441 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:18:07.441 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:18:07.452 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:18:07.453 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:18:07.498 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:18:07.569 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:18:07.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:18:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:18:07.673 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:18:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:18:07.706 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:18:07.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:18:07.725 INFO:teuthology.task.internal:Duration was 7356.003317 seconds 2024-03-25T18:18:07.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:18:07.757 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:18:07.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:18:07.760 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:18:07.803 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:18:07.806 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:18:08.018 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:18:08.018 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-03-25T18:18:08.019 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:18:08.038 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-03-25T18:18:08.039 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:18:08.065 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:18:08.065 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:18:08.081 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:18:08.134 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:18:08.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:18:08.150 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:18:08.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:18:08.287 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:18:08.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:18:08.314 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:18:08.342 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:18:08.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:18:08.357 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:18:08.380 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-03-25T18:18:08.407 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-03-25T18:18:08.440 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:18:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:18:08.465 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:18:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:18:08.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:18:08.506 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:18:08.509 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi084 2024-03-25T18:18:08.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:18:08.545 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255/remote/smithi157 2024-03-25T18:18:08.545 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:18:08.581 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:18:08.581 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:18:08.584 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:18:08.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:18:08.640 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:18:08.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:18:08.656 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:18:08.656 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:18:08.658 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:18:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:18 /home/ubuntu/cephtest 2024-03-25T18:18:08.685 INFO:teuthology.orchestra.run.smithi157.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:18 /home/ubuntu/cephtest 2024-03-25T18:18:08.686 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:18:08.696 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:18:08.799 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:18:08.826 DEBUG:teuthology.nuke:shortname: smithi084 2024-03-25T18:18:08.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:18:08.851 DEBUG:teuthology.nuke:shortname: smithi157 2024-03-25T18:18:08.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:18:08.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.932853', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3t1f7dXKRia+bTNuo77Ot0iB2dDzXvpcv7bpKqPxqen4vit2GWi0a+v/UoPo9CIZ+qZGjp4U2gYFz1A6NfDAI='} 2024-03-25T18:18:08.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.934352', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAW6YOCv5MC8vYhC9glGx5t7q8Pdb+sqj/Fviye5coELMdd8qf1QHkf4B+qZAVgJRcyOxlZUIQBMF9DZM5dI6a4='} 2024-03-25T18:18:08.907 INFO:teuthology.orchestra.console.smithi084:Power off 2024-03-25T18:18:08.907 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:18:08.924 INFO:teuthology.orchestra.console.smithi157:Power off 2024-03-25T18:18:08.924 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:18:08.933 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-03-25T18:18:08.934 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:08.946 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2024-03-25T18:18:08.947 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:08.956 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-25T18:18:08.970 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-25T18:18:12.957 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:12.972 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:13.069 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-25T18:18:13.082 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-25T18:18:17.074 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:17.083 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:17.185 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-25T18:18:17.193 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2024-03-25T18:18:17.294 INFO:teuthology.orchestra.console.smithi157:Power off completed 2024-03-25T18:18:21.190 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:18:21.300 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-03-25T18:18:21.401 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-03-25T18:18:21.505 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7356.0033168792725 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6455f336aa5e4253acd336102057eee5 status: fail success: false 2024-03-25T18:18:21.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:18:21.581 INFO:teuthology.run:FAIL